icon-search
markus-spiske-445253-unsplash

Type safe JSON parsing

Maciek Fulara 11.01.2017

Introduction

Problems with JSON payloads

A common problem when developing applications talking to external services are payloads that do not stick to the contract. When we build a typescript application consuming JSON endpoints the easiest thing to do (shown in most tutorials) is casting the payload to the expected type. This can however lead to hard to debug errors if the shape of the response returned by the endpoint changes. In this article we will build a simple mechanism for type safe JSON parsing.

Ways of dealing with JSON payloads

When you receive a JSON object in your application (ie. from an API call) there are a couple ways you can go about using it.

  • You could use it as is. The obvious downside of this approach is that the object will have the Any type so you cannot take advantage of the tooling support. You won’t get autocomplete, misspelled properties won’t fail the compilation, when you rename things you will have to rely on grepping the names etc.
  • You could cast the JSON object to the expected type. This is already better because you get tooling support but in case your cast isn’t correct you will get errors in arbitrary places of the application, wherever the object is used. Because of this it might not be immediately obvious what their cause is. This sort of errors are especially likely to happen if your JSON comes from an external API over which you have no control.
  • You could provide functions to validate JSON before you cast it to the expected type. This solves the problem with blindly casting JSON – if the shape does not match we get an error right where JSON is deserialized (possibly with a descriptive message – since we know here what the root cause of the problem is). This makes debugging easy but seems very repetitive and laborious – we have to write a validation for every type we deserialize from JSON.
  • Finally you could improve on the previous approach by decoupling the predicates that the object has to meet from the mechanism for checking them. This way once you have built a generic mechanism for verifying your predicates all you need to do when adding a new type is annotating it with the predicates. That is what we are going to build in this post.

Parser

In this post we are going to build a simple declarative JSON parser using typescript decorators.
Our toy parser will only have the most basic capabilities necessary to illustrate the concept.

  • the parser will not support more advanced typescript type construct like generics, union types etc.
  • the parser will only use constructor to initialize the object, it won’t set any properties on the object directly

Field decorator

We’ll start building our parser by creating a domain class to represent the metadata that we want to store with constructor arguments and a constructor parameter decorator. (If you are not familiar with decorators you can check our previous post).

export class ParamMetadata {
    constructor(public fieldName: string, public optional: boolean) {}
}

export function Field(fieldName: string, optional = false) {
    return (target: Object, propertyKey: string | symbol, parameterIndex: number) => {
        const paramMetadata: ParamMetadata[] = Reflect.getOwnMetadata("paramMetadata", target) || [];
        paramMetadata[parameterIndex] = new ParamMetadata(fieldName, optional);
        Reflect.defineMetadata("paramMetadata", paramMetadata, target);
    };
} 

Parser code

Next we’ll write the parser itself. We’ll start by showing you the whole parser first and than dissecting it bit by bit to see how it’s working.


import * as _ from "lodash";

export class JSONParser {

    public parse<T>(type: RefType<T>, json: Object): T {
        return <T>this.parseCompoundType(type, json, new Path().push("."));
    }

    private doParse(param: ParamTypeAndMetadata, json: Object, path: Path): any {
        const fieldName = param.paramMetadata.fieldName,
            isOptional = param.paramMetadata.optional,
            type = param.type;
        const value = json[fieldName];
        path = path.push(fieldName);

        if (!isOptional && _.isUndefined(value)) {
            throw new JSONParsingError("Missing required field", path);
        }

        if (this.isSimpleType(type)) {
            return this.parseSimpleType(type, value, path);
        } else {
            return this.parseCompoundType(type, value, path);
        }
    }

    private parseCompoundType(type: AnyRefType, json: Object, path: Path): any {
        const constructorParams = this.getParams(type);
        const constructorParamValues = constructorParams.map(param => this.doParse(param, json, path));
        return new type(... constructorParamValues);
    }

    private parseSimpleType<SimpleType>(type: AnyRefType, value: any, path: Path): SimpleType {
        const typeName = this.getTypeName(type);
        switch (typeName) {
            case "String":
                if (!_.isString(value)) {
                    throw new JSONParsingError("Expected String value", path);
                }
                return value;
            case "Number":
                if (!_.isNumber(value)) {
                   throw new JSONParsingError("Expected Number value", path);
                }
                return value;
            case "Boolean":
                if (!_.isNumber(value)) {
                   throw new JSONParsingError("Expected Boolean value", path);
                }
                return value;
            case "Date":
                if (!_.isDate(value)) {
                   throw new JSONParsingError("Expected Date value", path);
                }
                return value;
            default:
                throw new Error(`'${typeName} is not a member of SimpleType`);
        }
    }

    private getTypeName(type: AnyRefType) {
        if (type["name"]) {
            return type["name"];
        }
        const funcNameRegex = /function (.{1,})\(/,
            results = (funcNameRegex).exec(type.toString());
        return (results && results.length > 1) ? results[1] : "";
    }

    private isSimpleType(type: AnyRefType) {
        const simpleTypes = ["String", "Number", "Boolean", "Date"];
        return simpleTypes.indexOf(this.getTypeName(type)) > -1;
    }

    private getParams(type: AnyRefType): ParamTypeAndMetadata[] {
        const constructorParamTypes:  Array<AnyRefType> = Reflect.getMetadata("design:paramtypes", type),
            paramMetadata: ParamMetadata[] = Reflect.getMetadata("paramMetadata", type);
        if (!constructorParamTypes || !constructorParamTypes.length) {
            throw new JSONParsingError("Constructor has no arguments");
        }
        if (!paramMetadata || !paramMetadata.length) {
            throw new JSONParsingError("Constructor has no @Field annotatated arguments");
        }
        if (constructorParamTypes.length > paramMetadata.length) {
            throw new JSONParsingError("Constructor has arguments without @Field annotation");
        }
        return <ParamTypeAndMetadata[]>_.zipWith(constructorParamTypes, paramMetadata, (c, f) => new ParamTypeAndMetadata(c, f));
    }
}

export class ParamTypeAndMetadata{
     constructor(public type: AnyRefType, public paramMetadata: ParamMetadata) {}
}

export type AnyRefType = RefType<any>;

export interface RefType<T> extends Function {
    new(...args: any[]): T;
}

export class Path {
    private path: string[] = [];

    public push(fieldName: string) {
        const copy = new Path();
        copy.path = this.path.slice();
        copy.path.push(fieldName);
        return copy;
    }

    public toString(): string {
        return this.path.join("/");
    }
}

parse function

The parser has one entry point:

    public parse<T>(type: RefType<T>, json: Object): T {
        return <T>this.parseCompoundType(type, json, new Path().push("."));
    }
    export interface RefType<T> extends Function {
	new(...args: any[]): T;
    }

parse accepts a constructor function for type T and JSON data. It invokes parseCompoundType and casts the result to T.
parseCompoundType accepts one additional parameter of type Path. The JSON being parsed may have a nested structure – the path variable is a wrapper around String[] that is threaded through recursive calls to the parse functions and maintains the current path in the JSON tree. It’s used to produce meaningful messages pointing to the offending node in JSON tree if the parsing fails.

The following class definition:

class Person {
	constructor(@Field("name") public name: string, @Field("mother", true) public mother: Person) {}
}

and JSON:

{
	name: "a"
	mother: {
		name: "b"
		mother: {
			name: 1
		}
	}
}

will not pass validation. Thanks to the info accumulated in the path variable we can print the location of the offending field (“./mother/mother/name”)

parseSimpleType and parseCompoundType

Next we make a distinction between ‘simple’ types (String, Number, Boolean and Date) and ‘compound’ types (user defined classes which can be built from ‘simple’ types and ‘compound’ types).
We have two separate functions to parse ‘simple’ and ‘compound’ types: parseSimpleType and parseCompoundType.

parseSimpleType function

    private parseSimpleType<SimpleType>(type: AnyRefType, value: any, path: Path): SimpleType {
        const typeName = this.getTypeName(type);
        switch (typeName) {
            case "String":
                if (!_.isString(value)) {
                    throw new JSONParsingError("Expected String value", path);
                }
                return value;
            case "Number":
                if (!_.isNumber(value)) {
                   throw new JSONParsingError("Expected Number value", path);
                }
                return value;
            case "Boolean":
                if (!_.isNumber(value)) {
                   throw new JSONParsingError("Expected Boolean value", path);
                }
                return value;
            case "Date":
                if (!_.isDate(value)) {
                   throw new JSONParsingError("Expected Date value", path);
                }
                return value;
            default:
                throw new Error(`'${typeName} is not a member of SimpleType`);
        }
    }

parseSimpleType accepts a constructor, a value and a path. It then validates the type of the value based on constructor function’s name and if the validation fails raises an error. This is the only place where type validation takes place.

parseCompoundType function

This function is slightly more involved then parseSimpleType

    private parseCompoundType(type: AnyRefType, json: Object, path: Path): any {
        const constructorParams = this.getParams(type);
        const constructorParamValues = constructorParams.map(param => this.doParse(param, json, path));
        return new type(... constructorParamValues);
    }

    private getParams(type: AnyRefType): ParamTypeAndMetadata[] {
        const constructorParamTypes:  AnyRefType[] = Reflect.getMetadata("design:paramtypes", type),
            paramMetadata: ParamMetadata[] = Reflect.getMetadata("paramMetadata", type);
        if (!constructorParamTypes || !constructorParamTypes.length) {
            throw new JSONParsingError("Constructor has no arguments");
        }
        if (!paramMetadata || !paramMetadata.length) {
            throw new JSONParsingError("Constructor has no @Field annotatated arguments");
        }
        if (constructorParamTypes.length > paramMetadata.length) {
            throw new JSONParsingError("Constructor has arguments without @Field annotation");
        }
        return <ParamTypeAndMetadata[]>_.zipWith(constructorParamTypes, paramMetadata, (c, f) => new ParamTypeAndMetadata(c, f));
    }

    export class ParamTypeAndMetadata{
        constructor(public type: AnyRefType, public paramMetadata: ParamMetadata) {}
    }

In order to parse a ‘compound’ type we need to know a few things about its constructor arguments. For every argument we need to know what its type is, how does it map to JSON fields and whether it’s optional or not. We’ll use ParamTypeAndMetadata to encapsulate this data.

We use getParams helper function to get ParamTypeAndMetadata for type.

We get ParamMetadata array by retrieving "paramMetadata" that was stored by the Field decorator.

We get type information from "design:paramtypes". This metadata is written by the typescript compiler when you enable “emitDecoratorMetadata”. Typescript offers a basic reflection mechanism – it does not have support for more advanced type constructs – ie. if you had a union type the "design:paramtypes" array entry for that argument would contain Object; if you had parameterized type the generic information would be lost (String[] would become Array). Therefore if we wanted to extend our parser beyond the basic types we could not rely on typescript to provide the type information at runtime, we would need to preserve it ourselves using a decorator.

We do some simple validation on "paramMetadata" (we check that a constructor accepting arguments is present and that every constructor argument is annotated with a @Field decorator).

Next we zip "paramMetadata" and "design:paramtypes" info into one array of ParamTypeAndMetadata.

With the ParamTypeAndMetadata (type, mapping to JSON, optionality) for every constructor argument and the JSON object in hand we proceed to parsing the arguments. Each argument can be either of ‘simple’ type or ‘compound’ type. We have a separate function (doParse) that does the check and either calls parseSimpleType or recurs back to parseCompoundType.

Once we have the values for every argument we create and return a new object.

doParse function

Finally we have doParse

    private doParse(param: ParamTypeAndMetadata, json: Object, path: Path): any {
        const fieldName = param.paramMetadata.fieldName,
            isOptional = param.paramMetadata.optional,
            type = param.type;
        const value = json[fieldName];
        path = path.push(fieldName);

        if (!isOptional && _.isUndefined(value)) {
            throw new JSONParsingError("Missing required field", path);
        }

        if (this.isSimpleType(type)) {
            return this.parseSimpleType(type, value, path);
        } else {
            return this.parseCompoundType(type, value, path);
        }
    }

doParse is really straightforward – it updates the path (as we’re going one level deeper in the JSON tree), validates that the value is defined if it’s not optional and delegates the real parsing to parseSimpleType or parseCompoundType.

Summary

The parser that we’ve built is very simple but could easily be extended to be used in production settings. In one of our projects at Sparkbit we have a JSON parser based on the same principles that can validate generic types and do simple conversions before parsing (ie. some of the APIs the project is using return strings where numbers are expected).

We hope you found this article useful.

comments: 0