The Road Onwards - Confession 46

2014.12.28 23:56:57

header Just yesterday I finally finished the last big problem that plagued Radiance since its conception. Actually, the issue dates even further back all the way to when I started TyNET, since I never really did solve it until now. Though the main reason for this long delay is that I wasn't completely conscious of the issue until about half a year ago. For better or worse, things worked just well enough without absolutely having to solve it. But, ever since I grew aware of it it has been a thorn in my side as I couldn't figure out how to elegantly solve it.

The problem is what I've touched upon for a bit many a time before, but it is so closely intertwined with other parts of the system that it's hard to talk about it in brief. I will try to explain it in an as straight-forward manner as I can muster.

I'm first going to start out by explaining one of Radiance's design goals. From the get go, one of the things that seemed important to me even in previous versions of TyNET was that whatever the end-user might want to run on their server would have to manage being usable even if it were constrained to a certain path, or without subdomains to use for module addressing. As such, it has to be possible to somehow squeeze the existing model into one that has potentially very different paths for everything, without requiring too much user intervention.

Offering a solution to this problem pushes us into the following conundrum: How can we write anything at all if we never really know where any of our pages or resources will be? And, well, the only way to make this possible is to create a virtual address space. This means that anything you do when working on your module will live in this agnostic Radiance world. Handling the discrepancies between the outside world and our virtual one however has to be performed flawlessly by the framework in order for this to work.

This virtual space then has strong influence on how requests are handled and how resources are addressed. After all, the server will deal with the outside world and has to figure out how to translate that to the virtual equivalent. Similarly, resources that the user has to have relative access to –such as style sheets and images– need to be translated back from the internal format to something that makes sense in userland.

At the same time this all is complicated by other Radiance design principles: Keep it simple for the author and offer choice where possible. In more concrete terms, the latter means that the writer of a module should not necessarily be constrained by framework design decisions if possible. The framework should act as a guidance and toolkit for building applications, not as a rigid system that enforces strong rules. So, it needs to be possible for modules to operate without falling into the complications set forth by the divide of the two worlds. Similarly Radiance should provide means to make it easy to employ this system if the author so desires.

The first part to solving everything is Radiance's URI object, which is loosely modelled after the URL. It is made up of a list of domains, an optional port, and a path. URIs can be compared for equality and precedence, and can be translated back to an URL. They can also be transformed into internal or external equivalents, which represent resources in either worlds.

Transformation of URIs is achieved through the routing system, which consists of a mapping and reversal part. Routes in the context of Radiance do not describe the same mechanism as is usually associated with the term in other Frameworks; they don't deal with matching an address with a page, but instead perform arbitrary transformations on the URI objects to provide a bridge between the worlds.

This solves our problem nicely. At least in theory. In practise things get more complicated when we take base domains and absence of requests into account: On any system you will have a base hostname that always remains on any request, that much is a given. In order for the system to work, these domain parts need to be cut off properly and re-attached on reversal. Due to the ambiguity in domain names and domain spaces, the framework cannot figure this out automatically, and so the user will have to specify all the domains that he intends to use the framework on in the configuration (something I would have liked to avoid). Due to multiple domains being possible, which one is currently used for any particular request needs to be carried with the request data. This in turn causes a problem when data needs to be generated outside of requests (such as when cache is revalidated). The system somehow needs to know the proper domain to use to generate the right reversal. For this, I still do not know a proper fix. Currently it just picks the first option from the configuration, but that won't always be the right thing to do.

So, with some special care taken for base domains and requests, the URI+Routing system actually solves the problem. The last part, ease of use, is in part provided by patterns and the extensions to clip that r-clip provides. Templates that use clip can profit from having a direct syntax for Patterns that will be resolved according to proper resource mechanisms and URI transformations. Adding support for similar mechanisms to other templating systems should not be hard.

To close off, I'll go step by step through the standard life cycle of a request as it goes through the Radiance system. First, the server implementation accepts an HTTP request and parses that. It sends all the relevant data such as the pure request URI, headers, post, get, cookies, etc. along to Radiance's request. This in turn constructs a request and response object and transforms the URI into its internal representation using the routing system with internal-uri (this also determines the currently active domain and saves it in the request object). Once the request and response objects are bound to their corresponding special variables, the URI is passed along to dispatch. This performs a series of URI matching tests, all according to URI priority and precedence. Once a match occurs, the associated dispatcher function is executed. This will usually be a page or an API endpoint and may perform any further environment processing actions before the user code is executed. Within the user page most likely some kind of data will be rendered. In case a URI is being rendered, it will (again, probably) be passed along to uri-to-url which in turn calls represent-uri whereupon the URI will be sent to external-uri to produce a matching part that is then rendered to a usable URL for the browser. This is then put into the output in some fashion. The user may either set the data field of the response object, or simply return its response. The stack unwinds all the way down to request, where the output data is checked for type conformity and then finally passed down to the server implementation that will have to figure out how to send it all back to the user.

Important to notice for this walkthrough is that none of the functions involved are particularly bound in any fashion. request can be called from the REPL with made-up data and it should still work, the same goes for the routing and URI related functions and even for dispatch (though that might fail if the pages expect an existing request object). The prevalent idea hereby always being that by keeping things separate it will be easier for the user to take the parts they need and put them together in the way they want. It would be perfectly legal for a server implementation to completely bypass the request and handle that all on its own somehow, etc.

I'm actually not sure how much of this was very comprehensible for anyone but me. Thinking about systems for long times tends to skew your impressions about what's obvious and what isn't. I'll probably have to figure out a better way to explain this all for the documentation, whenever I inevitably get to that.

For what it's worth, Radiance is now pretty much done, not accounting for whatever changes will have to be made whenever I go through the specification process. Until then, I'm just happy to finally put my mind to rest on this issue.

Written by shinmera