I spent a year of my life making an ASN.1 compiler in D

Utilise metaprogramming so your compiler can stay dumb/poorly made

Templates can provide really natural APIs while still catching errors at compile time

D Snark: The forever-experimental allocator package

I can’t retain all the information I need to know

This post is unstructured, so you can just pick somewhere random and start reading from there with no/minimal context lost.

I’m currently writing Juptune – a toy async I/O framework that attempts to implement as much of its stack as possible in pure D.

I’m really interested in writing an implementation of TLS, which means I need to be able to handle x.509 certificates (i.e. TLS/SSL certs), which means I need to be able to handle their underlying data encoding: ASN.1’s DER encoding.

So basically I just wanted to do this for fun at the end of the day, nothing much deeper than that. I’ve never written or worked on a proper compiler project before that wasn’t toy-sized so I saw a ton of growth potential… the main thing that’s grown however is the mental scar ASN.1’s left on me.

I’ve succesfully generated code that can parse a couple of x.509 certificates I’ve thrown at it, and I’ve started work on an almost-D-native (excluding crypto primitives) implementation of TLS 1.3.

ASN.1 is the result of a bunch of graybeards from the late 80s+ trying to design an overengineered data specification language. In other words, it’s protobuf on steroids.

There’s two parts of ASN.1: There’s the ASN.1 notation (defined by x.680, x.681, x.682, and x.683), and then there’s the various encodings (BER, CER, DER, PER, XER, JER…). In this post I’ll mainly be focusing on the notation + DER.

Similarly to protobuf you use the notation to define a structured way to represent data, and then use tooling that can generate encoders/decoders for a specific encoding, in a specific programming language.

Here’s a choice snippet of the ASN.1 notation for RFC 5280 (which defines what’s commonly known as TLS certificates):

Did I ever mention that ASN.1 is complicated? On the one hand the sheer amount of possible encodings is daunting, but on the other hand it shows a certain flexbility that ASN.1 provides – you could even invent your own domain-specific encoding if needed.

Loosely speaking you can define ASN.1’s notation as being the “base” notation defined in x.680, with the sometimes-optional addon specifications defined in x.681, x.682, x.683.

These specifications are also written in academicese so for mere uneducated mortals such as myself, simply trying to read and understand what the specifications are saying in the first place is already a large hurdle. I think I’ve started to get the hang of it though.

Fortunately for my use case of handling x.509 certificates, there’s no hard requirement for anything beyond x.680 and so x.680 is the only spec I’ve attempted to implement so far (outside of x.690 which describes how BER/CER/DER works – which is actually a joy to read compared to the x.68x specs).

x.680 isn’t the worst thing in the world to implement, it’s just the fact that there’s a lot more to it than you’d think from a quick glance at a code example, as well as some relatively annoying “transformation” (semantic) rules you have to acccount for.

One of the more annoying parts of implementing a parser for ASN.1’s notation is that x.680 has been revised several times over the years, which includes the deprecation + removal of certain features.

Meaning that if you want to write a compiler for ASN.1 for a specific use case, but want it to also be an implementation of the more modern specs… then you’ll have to partially implement/hack around some of the older stuff that’s no longer defined in the up to date spec documentation.

This is essentially the academic equivalent of an Elder Scroll – you will go insane attempting to read let alone mentally parse this damn thing.

I’d absolutely love to attempt to implement x.681 for the challenge of this feature alone, however I only have so much energy (and sanity), so it’ll likely be a while until I even properly consider it.

x.683 describes the ability to create templated (sorry, “parameterised”) types. Similar to the other ASN.1 extensions I haven’t looked much into this feature, but it appears to be a lot simpler to implement than the others.

MyTemplatedThingy{ValueT} ::= SEQUENCE { value ValueT} MyStringThingy ::= MyTemplatedThingy{UTF8String}

It supports values as well as types within its template parameters (similarly to D!) so there’s a few cool things you can do with it I guess.

Despite the many, many, many pains of this god forsaken technology, it’s actually really interesting and powerful at the same time.

ASN.1’s notation contains a pretty neat feature where you can add special constraints onto types + fields. So rather than having a stray “ProtcolPacket.field1.field2.xyz MUST be between 0 and 2” that’s super easy to miss, you can instead describe this constraint within ASN.1 itself which (good) tooling will then take into account for you.

It’s really cool to see that ASN.1 has a feature like this though, considering the only other langauge I’ve personally encountered that has a similar feature is Ada.

ASN.1 generally uses the OBJECT IDENTIFIER type in order to, well, identify specific things, e.g. extensions found within x.509 certificates.

PKIX1Implicit88 { iso(1) identified-organization(3) dod(6) internet(1) security(5) mechanisms(5) pkix(7) id-mod(0) id-pkix1-implicit(19)} DEFINITIONS IMPLICIT TAGS ::= BEGIN — .. — END

Everything between the curly brackets is an OBJECT IDENTIFIER for this exact module – technically no other ASN.1 module in existance should ever use this specific OBJECT IDENTIFIER. The optional labels (e.g. iso) have no meaning beyond aiding human comprehension, it’s the values (e.g. (0)) that are actually used to create the identifier.

PKIX1Implicit-2009 {iso(1) identified-organization(3) dod(6) internet(1) security(5) mechanisms(5) pkix(7) id-mod(0) id-mod-pkix1-implicit-02(59)}DEFINITIONS IMPLICIT TAGS ::= BEGIN — .. — END

This updated version doesn’t change how data is encoded to/from DER but instead it simply uses more modern syntax and features.

When importing modules within ASN.1 notation you can (and should) specify an OBJECT IDENTIFIER as well:

Maybe I’m just a nerd, but I find this to almost be a thing of beauty with how simple yet effective it is.

D has several quality of life features that makes it surprisingly easy to generate code for – features that would definitely make the compiler more annoying to work with when targeting other languages.

These features on their own aren’t exactly rare to see, but the specific combination is what makes everything work together so well.

static import in D means “import this module, but ONLY allow it to be used via its fully qualified name”:

This feature is a godsend for preserving the original names of ASN.1 types. For example, Juptune provides an error type called Result which comes from the juptune.core.util.result module.

Without static imports I’d have to be careful of ASN.1 code that defines a Result type as it’d otherwise come into conflict with Juptune’s own Result type.

static import jres = juptune.core.util.result; // From ASN.1 definition: Result ::= SEQUENCE { — yada yada — }struct Result{ jres.Result set(/*…*/) @nogc nothrow { /*…*/ return jres.Result.noError; }}

On a similar vein D allows you to specify that instead of looking up a symbol from any available symbol table (e.g. local vars; non-static imports, etc.) it should instead perform a lookup using the current module’s top-level symbols.

/* Given this ASN.1 notation: Type1 ::= SEQUENCE { — yada yada — } Type2 ::= SEQUENCE { type2 Type2 }*/ // The following types are generated struct Type1 { /* yada yada */ } struct Type2{ private { .Type1 _type1; }}

Essentially, this feature compliments the static import feature to help make it much harder for ASN.1 types to accidentally refer to the wrong symbol when converted into D code.

First example is around how some getters and setters for SEQEUENCE fields are generated. Instead of doing the correct thing and preserving the type name for each field, I got lazy and just used typeof(_field):

// Heavily omitted examplestruct Dss_Parms{ private { asn1.Asn1Integer _p; } jres.Result setP(typeof(_p) value) @nogc nothrow { /* .. */ } typeof(_p) getP() @nogc nothrow { /* .. */ }}

The second example is around error messages. Instead of needing to keep track of the current type’s name when generating error messages… I could just use typeof(this) to get the type instead:

// Heavily omitted examplestruct Dss_Parms{ jres.Result fromDecoding(/* .. */) @nogc { /* .. */ result = asn1.asn1DecodeComponentHeader!ruleset(memory, componentHeader); if(result.isError) { return result.wrapError( “when decoding header of field ‘p’ in type ” ~__traits(identifier, typeof(this)) ~”:” ); } /* .. */ }}

What’s even better is that because the entire string is composed of compile-time constants, it doesn’t actually require an allocation + concat at runtime since the compiler will constant fold it for you. This allows fromDecoding to still be marked as @nogc!

// Heavily omittedstruct Dss_Parms{ void toString(SinkT)(scope SinkT sink, int depth = 0,) { /* .. */ static if(__traits(hasMember, typeof(_p), “toString”)) _p.toString(sink, depth+1); else { putIndent(); sink(“n”); } /* .. */ }}

You could definitely utilise D’s metaprogramming for more complicated stuff, but it’s also good for silly little things like this.

Naturally I’ve tried to use whatever D features that I could in order to implement dasn1, so I thought I’d pick a few parts of the code that rely on D’s features quite heavily as a small showcase.