Fengyun Liu

Why Every Complex System is in Need of a DSL

10 Jan 2022

Almost all complex systems are bound with one or more languages that are different from the main language used for developing the system. Here are some examples:

  • Web browsers initially can only render static HTML pages. Soon, JavaScript was introduced to make web pages more interactive.

  • Nearly all database systems are equipped with a query language as the interface language for data manipulation. For example, SQL is used in relational databases.

  • The core technology underpinning PDF is the language PostScript.

  • IDEs usually support extensions, and the extensions are written in another language. For example, Lisp for Emacs and JavaScript for Visual Studio Code.

  • Nearly all operating systems support managing the OS via shell script languages. Meanwhile, most operating system also support extending kernels via Berkeley Packet Filter.

  • Spreadsheet software usually enable users to create relationships between cells in a domain-specific language, e.g., VBScript for Excel.

  • Cloud applications usually support vendor customization through a domain-specific language. For example, Shopify relies on Liquid for defining customized stores.

  • The language Lua is widely used in game development.

  • Graphical systems usually support shading languages for specifying shading effects, such as texture, lighting, etc.

  • The most widely used document format PDF is based on PostScript, no mention the typesetting system TeX.

For simplicity, let’s abuse terms and call these languages domain-specific languages (DSL), in spite of the fact that some of them are obviously general-purpose languages, such as Lisp and JavaScript. The DSLs usually allow users to partly control the look and behaviors of the underlying systems. We may classify the usage into several categories:

  • Language as the main interface for accessing services provided by the system. That is the case for SQL, shading languages and shell scripts.

  • Language to customize the look and/or behaviors of the underlying system, e.g., Liquid to customize the theme of Shopify stores.

  • Language to enhance the system capabilities, e.g., BPF to easily extend and customize the kernel, and Lisp to extend and customize Emacs.

While whether some of human linguistic capabilities are innate is still open to debate, there is no doubt that the ability to learn and use languages is one of our most important asset. Language as text is easy to store, transform and share. A language interface is more flexible and powerful than any fixed graphical interfaces.

The downside of using a language as interface is the learning curve, memory burden and composition overhead. However, interactive graphical interfaces can be created to synthesize the code for simple and common use cases, which avoids the drawbacks. Meanwhile, the language interface still makes it possible to support complex use cases.

Simple customization can be implemented as configuration options. However, for complex customization, e.g., vendor-specific discount logic of a multi-tenant sales system, a language cannot be avoided.

As cloud-based application is becoming more popular, I conjecture the need for tenant-based customization will grow as well and so will be the need for DSLs to support such customization.

Extending the capabilities of the underlying system is an eternal need in software development. For a complex system that intends to be used for decades, the ability to add new capabilities without sacrificing system reliability is important. One flexible way to achieve the goal is to support an extension system. The extension system enables system modules to be developed in a light-weight process. There is no need for the code of extensions to be merged with the code of the core system. Conceptually, it’s not necessary for the extensions to be developed in a DSL. However, a DSL may boost productivity of programmers, enjoy language-based security and defend against runtime errors by the means of type safety or static analysis.

To summarize, the usage of DSLs boils down to three fundamental requirements in complex systems:

  • Interfacing
  • Customization
  • Extension

Now, can you name a complex system that does not have the needs listed above?

There are many concerns regarding the design of DSLs for complex systems:

  • usability
  • productivity
  • expressiveness
  • security
  • safety

I’ll leave them for another post.