> oh. did i meet you there?

Probably :-). Ron was a huge fan of Roger Needham's (and, ngl, I was too) and Roger along with Andy Birrell and others were on a kick to make RPCs "seamless" so that you could reason about them like you did computer programs that were all local. Roger and I debated whether or not it was achievable (vs. desirable) at Cambridge when we had the PKI meeting there. We both agreed that computers would get fast and cheap enough that the value of having a canonical form on the wire vastly outweighed any disadvantage that "some" clients would have to conversion to put things in a native format they understood. (Andy wasn't convinced of that, at least at that time). But I believe that was the principle behind the insistence on ASN.1, determinism and canonical formats. Once you built the marshalling/unmarshalling libraries you could treat them as a constant tax on latency. That made analyzing state machines easier and debugging race conditions. Plus when they improved you could just replace the constants you used for the time it would take.

XDR is like a four-octet aligned version of PER for a cut-down version of ASN.1. It's really neat.

XDR would not need much work to be a full-blown ER for ASN.1... But XDR is extremely inefficient as to booleans (4 bytes per!) and optional fields (since they are encoded as a 4-byte boolean followed by the value if the field is present).

You can thank Tom Lyon for it. Tom pretty much did the entire RPC/XDR/NFS stack to kick things off.

That matches the advice from Latacora[1]. That advice makes a lot of sense from a security correctness and surface area perspective.

There's a potential developer experience and efficiency concern, though. This likely forces two deserialization operations, and therefore two big memory copies, once for deserializing the envelope and once for deserializing the inner message. If we assume that most of the outer message is the inner message, and relatively little of it is the signature or MAC, then our extra memory copy is for almost the full length of the full message.

[1]: https://www.latacora.com/blog/2019/07/24/how-not-to/

Correct, the compiler allows you to generate the Rust bindings automatically. Worth noting that the compiler is at an earlier stage of development (the library was started six years ago, the compiler started roughly two years ago). So there are features that aren't used or supported by the compiler that are available in the library.

Yes writing the definitions by hand can time consuming and error-prone, but I designed the library in mind around the declarative API to make it easy to both manually write and generate, I also personally prefer writing Rust whenever possible, so nowadays I would sooner write an ASN.1 module in Rust and then if needed build a generator for the ASN.1 textual representation than write ASN.1 directly since I get access to much better and stronger tooling.

Also in my research when designing the crate, there are often requests in other ASN.1 or X.509 libraries to allow decoding semantically invalid messages because in the wild there are often services sending incorrect data, and so I designed rasn to allow you mix and match and easily build your own ASN.1 types from definitions so that when you do need something bespoke, it's easy and safe.

The Python library proposed in TFA is to be based on a different, DER-only Rust ASN.1 library[1]. So ASN.1 in Rust is more than tangentially relevant here.

[1] https://github.com/alex/rust-asn1

Understanding prior art and getting more comprehensive list of things that need to be considered is always good.

Not doing it is like inventing new programming language after just learning one of them.

Would you rather they reinvent the wheel badly? Thjat's what ProtocolBuffers is: badly reinvented ASN.1/DER!

PB is:

  - TLV (tag-length-value), like DER
  - you have to explicitly list the
    tags in the IDL as if it was ASN.1
    in 1984 (but actually, worse,
    because even back then tags were
    not always required in ASN.1, only
    for diambiguation)
  - it's super similar to DER, yet not
    not the same
  - PB was created in part because ASN.1
    had so little open source tooling,
    but PB had none until they wrote it
    so they could just have written the
    ASN.1 tooling they'd wished they had

smh

What should people use today, given the choice, that isn't ASN.1?

Edited to add: If they need something with a canonical byte representation, for example for hashing or MAC purposes?

Any luck with it?

Check the README of: https://web.archive.org/web/20240416031004/https://github.co...

I need something like this.

No, it is not the same. Sadly the author probably made it private, might reach out to him, just don't know how. :( He did not put his e-mail anywhere from what I have found so far.

https://github.com/zhonghuihuo/asn1-tool is still available, but it is very old, it is probably a VERY old fork.

I need something like this :(. I need it for Java / Kotlin. I do not have the repository cloned, so I am kind of in the dark.[1]

Found the archived page of the previously mentioned project that is probably private now: https://web.archive.org/web/20240416031004/https://github.co...

[1] Never mind, I found the newest (probably) asn1-compiler.jar! I still need an actively maintained alternative, however, for Java / Kotlin. For ASN.1 between Erlang / Elixir <> Java / Kotlin.

The ASN.1 format itself isn't too bad. It shows its age and has some very weird decisions behind it as places, but it's not that difficult to encode and is quite efficient.

Unfortunately, the protocols themselves can be confusing, badly (or worse: inconsistently) documented, and the binary formats often lack sensible backwards compatibility (or, even worse: optional backwards compatibility). Definitions are spread across different protocols (and versions thereof) and vendors within the space like to make their own custom protocols that are like the official standardised protocols, but slightly different in infuriating ways.

If you parser works (something open source rarely cares about so good luck finding one for your platform), the definitions extracted from those DOCX files are probably the least of your challenges.

First you can download specifications in either PDF or doc(x). Second doc(x) are simple enough that simple doc(x) to ASCII/text is good enough to produce working ASN.1 definition. Copy&paste is also an option.

There are many tools that can handle UPER up to certain level (some rare ASN.1 types might not be supported). I think the main issue is not in the codec, rather the lack of compilers that can create correct language-level representation of the ASN.1 definitions. 3GPP specifications are enormous and you don't want to create them by hand. ASN.1 has some very difficult notations, e.g. inner subtype constraints and information object classes. Subtype constraints may affect for the encoding output in UPER and if you are not representing them correctly overall, then you are not compatible.

PER were defined in 1994; back then applying zlib wasn't something you "just" do. Modern use is backwards compatibility (or cargo cult.)

When every bit passing through the network gets charged (if not to the customer, then it's taking up capacity that could otherwise be charged to the customer), and the software in the endpoints needs to be as low-power as possible, zlib is additional overhead you definitely don't want.

Most of the open source tools need patching to properly support certain scenarios (been there done that). They also lack support for parsing ASN.1 Value Notation format (textual), which is used everywhere in specifications, OSS Nokalva offers the full set of tools to handle this even with a playground and ASN.1 editor, this is non-existent in open source right now. For now the open source tools only focus on the crypto aspect, and doesn't really dive into telco, banking, biometric, and others.

Basically any commercial ASN.1 compiler prevents usage of the output in any open-source project. There is that.

Meh. After all, if you're not using ASN.1 you're using something like ProtocolBuffers or FlatBuffers or whatever and all open source tooling.

Where is the monoculture here?

The asn1 crate provides three builtin Choice enums itself - asn1::{Choice1, Choice2, Choice3} - which support 1, 2 and 3 choices respectively. I assume that is what lilyball is referring to. But as you correctly point out the custom derive supports mapping enums with more variants to CHOICE just fine, so the builtin enums are not a relevant limitation.

I always thought the selling point of Pure Python was that you might be running on some esoteric implementation of python, or hardware that the library maintainer didn't include binaries for.

I mean, I am glad wheels exist, they make things a lot easier for a lot of people. I just believed in the dream of targeting the language for maximum compatibility and hoping a better implementation would eventually run it faster.

Why is it tragic? It's more or less idiomatic in Python to put the hot or performance-sensitive paths of a package in native code; Rust has arguably made that into a much safer practice.

It's part of the original selling points of python, so it's not surprising that we've never stopped doing it.

Yeah, I heard about this new one taken from browsers.

It is supposed to fix everything that previous ones never achieved.

In what way does having to compile a Rust library prevent this?