04 add controller

Recap: What is MVC?

In the previous stage you learned about the model-view-controller (MVC) approach to application architecture. MVC separates thinking about and working with the information content (the model) from thinking about and configuring the way it is presented (for example, as HTML). To make it easier to distinguish the TEI source data from the model from the eventual HTML (or other) view you started using a namespace for the model, which you bound to the m: namespace prefix, and in your tmp file in the previous stage you created HTML output in the HTML namespace, which you bound to the html: namespace prefix. Your TEI source files are, of course, in the TEI namespace, which you’ve bound to the tei: namespace prefix.

Namespaces make it possible to use the same local name for elements that are part of different schemas. For example, an element <p> in the TEI namespace (with the fully qualified name, or FQName, <tei:p>) is not the same as <p> in your model namespace (<m:p>), and the HTML element happens to have the same local element name, too (<html:p>). Why is it important for these elements to be differentiated? They all represent the same type of information—something we understand as a paragraph—but TEI paragraphs have different possible content from HTML paragraphs or model paragraphs. (As an alternative to namespaces it would be possible to use different element names, but namespaces leverage the similarity of the three types of paragraphs in a standardized, self-documenting way by giving them the same local name.)

If all mappings from TEI element to HTML element were as direct and straightforward as the one-to-one mapping of paragraphs you could omit a separate model level and transform directly from TEI to HTML, and you’ll do that where it’s useful later. Sometimes, though, the model contains structures that don’t have a one-to-one correspondence with the TEI. For example, what if you want to provide only the first few words of the first paragraph of each article as an incipit on the search results page? The beginning of the first paragraph isn’t a structural component of the TEI, but you can extract those words into an <m:incipit> element that you create in the model and then transform it to an HTML <span> or something similar in the view. If you want to render the whole first paragraph as a tool tip when the user hovers over the incipit, you could also map the first <p> of the TEI to an <m:p>. It isn’t surprising that some features of the model will correspond directly to features of the TEI sources, while others will be derived from the TEI but not in a one-to-one way, and still others might be created from scratch. Putting the source and the model in different namespaces provides a standardized way to customize the markup for each of the two purposes.

For most of this laboratory edition’s application features, then, you’ll want to write XQuery that transforms the TEI sources to an appropriate purpose-specific data model, which you’ll then transform to HTML. There are other transformations your edition might require, though. For example, you might want your XQuery to output not only HTML for reading, but also comma-separated values (CSV) you can use in Gephi for network analysis, plus SVG for graphic visualization, plus other formats for other purposes. The data from your TEI sources that supplies the information behind those outputs might be partly (or even entirely) the same, and in that context MVC architecture would let you separate selecting and organizing data for the output from formatting it for presentation. Visitors to your web application won’t know (or care) about whether the output is created all at once or in a pipeline of steps, but you, as the developer, will be able to focus on the subcomponents of the larger task individually, which can make for easier planning, implementation, and testing.

Goals for this stage

In MVC architecture the C stands for controller, which in an eXist-db app is an XQuery resource that receives all user requests and specifies how they should be understood and processed to create the model and then transform the model into the view. In the last session there was no controller because you used a single XQuery document to transform the TEI source to the model and then, in a separate step within the same XQuery file, the model to HTML. In this session you’re going to:

1. Create only the model in modules/titles.xql
2. Create a views subdirectory to hold the XQuery that will transform the model to HTML, which you’ll call views/titles-to-html.xql
3. Create a controller XQuery resource, called controller.xql, in the main directory of the app, which will ensure that the model and view can find each other
4. Confirm that the new modular implementation produces the desired HTML output. We’ve deliberately made a mistake in our implementation below, which we’ll fix as way of illustrating some useful debugging strategies for MVC development.

The controller 1) receives requests for information (when, for example, the user clicks on a link), 2) invokes the XQuery that creates the appropriate model, and then 3) passes that model through one or more XQuery transformations to create and return the view (in this case as HTML). Input from the user in your app will come from a web browser. We won’t go into detail about how the controller works at this point, but you can read more about it on p. 194 of the eXist-db book.

(Strictly speaking, the controller, and the app in general, don’t know whether the user clicked on a link, typed something into a browser address bar, or implemented an HTTP connection to eXist-db in some other way. You can reasonably expect, though, that users will employ the links you provide to interact with your app.)

Returning to your workstation and connecting to eXist-db

1. Start the eXist-db database. The easiest way to do this is to open the application on your desktop. While you will see some waiting-related screens, eXist-db won’t open a graphical user interface (GUI) like you would expect from something like a text editor. Instead, you can confirm that it is running by checking the widget it creates (either top or bottom tray, depending on your operating system) or by visiting http://localhost:8080 in a browser.
2. Start VS Code by opening the application. Open your project workspace inside VS Code. Once you’ve confirmed that the database is running, start synchronization between the workspace and the database by using either the command palette or the autosync button in the lower right hand corner of the editor (see the image below).

The synchronization ensures that changes you make to your files inside VS Code will be uploaded to your running instance of eXist-db each time you save within VS Code. If you forget to turn on synchronization, you can catch up by building the app (type ant at a command line), installing it into the eXist-db instance (using the eXist-db package manager), and then turning on synchronization in VS Code.

In VSCode …

1. Once you’ve confirmed that VS Code is synchronizing with a running eXist-db instance, open a shell and navigate to the main directory of your project. Use mkdir views to create a new subdirectory called views.
2. Inside VS Code create a new XQuery file inside the new views subdirectory called titles-to-html.xql. The naming convention in your app will be that modules/filename.xql and views/filename-to-html.xql will work together for any value of filename, which means that there will be one module file and one view file for each each type of query the user can make.
3. The new XQuery file needs to know about namespaces, so copy them from titles.xql. Your new file should look like the following so far:

xquery version "3.1";
(:==========
Declare namespaces
===========:)
declare namespace hoax = "http://www.obdurodon.org/hoaxed";
declare namespace m = "http://www.obdurodon.org/model";
declare namespace tei = "http://www.tei-c.org/ns/1.0";
declare namespace html="http://www.w3.org/1999/xhtml";

4. Now find the bit of XQuery you wrote in the previous stage to transform the model to HTML and move it into the new file. You can find the snippet you’re looking for in the 03-tmp-HTML branch in the file titles-html.xql, where it looks like:

<html:section>
    <html:ul>{ 
        for $item in $data/m:item
        return
            <html:li>{$item/m:title || ", " || $item/m:date}</html:li>
    }</html:ul>
</html:section>

So far, all you’ve done is copy code from one file to another. You won’t use tmp/titles-html.xql any more, so you can delete it once you’ve completed the copying. Originally the for loop that creates the <html:li> items used a variable called $data that you declared in the same file. Since you aren’t declaring $data yet in the new views/titles-to-html.xql file, you’ll see a warning that “variable $data is not set”. You’re still creating the model, which is the value of $data, in modules/titles.xql, and eventually the controller will pass it into the new file, but you can tell the new file to accept the passed content by adding the following after the namespace declarations:

declare variable $data as document-node() := request:get-data();

The function request:get-data() retrieves the model, which the controller will pass along from modules/titles.xql, and make it available in a variable called $data. Since the model is an interim document, you want the variable to represent a document node (rather than an XML element or a string, for example). (We’ve deliberately introduced a mistake here, which we’ll debug below. Stay tuned!)

The XQuery that creates the view can now retrieve the model, but without a controller to forward the model to the new XQuery there’s nothing to retrieve. Below is the full controller file, which is also available in the repository from the Stage 04 branch onwards. You should copy and save it to your main project directory as controller.xql:

xquery version "3.1";
declare namespace html="http://www.w3.org/1999/xhtml";

declare variable $exist:root external;
declare variable $exist:prefix external;
declare variable $exist:controller external;
declare variable $exist:path external;
declare variable $exist:resource external;

declare variable $uri as xs:anyURI := request:get-uri();
declare variable $context as xs:string := request:get-context-path();
declare variable $ftcontroller as xs:string := concat($context, $exist:prefix, $exist:controller, '/');

if ($exist:resource eq '') then
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
        <redirect url="index"/>
    </dispatch>
else
    if (not(contains($exist:resource, '.')))
    then
        <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
            <forward url="{concat($exist:controller, '/modules', $exist:path, '.xql')}">
                <add-parameter name="exist:root" value="{$exist:root}"/>
                <add-parameter name="exist:controller" value="{$exist:controller}"/>
                <add-parameter name="exist:prefix" value="{$exist:prefix}"/>
            </forward>
            <view>
                (:transformation to html is different for different modules:)
                <forward url="{concat($exist:controller, '/views/', $exist:path, '-to-html.xql')}"/>
            </view>
            <cache-control cache="no"/>
        </dispatch>
    else
        <ignore xmlns="http://exist.sourceforge.net/NS/exist">
            <cache-control cache="yes"/>
        </ignore>

Now that your app has a controller, let’s break it down into human language. You declare some variables, some of which are included by default in your eXist-db instance, while others are strings or URLs that you fetch or construct using functions. We’ll talk about how to use those later. Most of the work of the controller happens in an if/then/else statement that evaluates a URL and acts in different ways according to whether the URL is empty, is present but does not contain a dot, or contains a dot. Here are the details:

• If the query hasn’t asked for a page by filename ($exist:resource eq ''), the controller will redirect the user to index, a main page that you’ll add to the app at a later stage.
• If there is a filename and it doesn't contain a dot ((if (not(contains($exist:resource, '.')))), then you forward the request to a URL you construct. The new URL you construct is {your app’s base URL}/modules/{user input}.xql. This means that if you go to our browser and type the app’s base URL plus /titles, the controller will look in the modules subdirectory for a file with a name formed by appending the filename extension .xql to the request. For example, if your app is called hoaXed and you enter http://localhost:8080/exist/apps/hoaXed/titles into the browser address bar, the controller will access and execute the modules/titles.xql file you just wrote. As part of making that request, the controller passes some parameters (named values), and we’ll say more about those later. modules/titles.xql creates the model, and the <view> part of the controller then forwards the model to the XQuery that will transform it into the view. This forwarding of the output of modules/titles.xql into the inputof views/titles-to-html.xql is the API POST call we mentioned earlier. It constructs the name of the XQuery that creates the view by looking in the views subdirectory and appending the string -to-html.xql to the name requested in the original query.
• The final else handles non-XQuery files, such as CSS and images, all of which will contain a dot before the filename extension. The controller receives all requests; the preceding clause assumed that requests without a dot were for XQuery files in the modules subdirectory, and this clause says that anything with a dot in it (that is, with a filename extension) should be returned exactly as requested. Users won’t request CSS or image files directly, but the HTML constructed for the view might link to those files, so the controller needs to know to return them without rewriting the URL.

When users ask for a URL like http://localhost:8080/exist/apps/hoaXed/titles, they specify one file by name (titles with no preceding subdirectory and no filename extension) and the controller transforms the request to modules/titles.xql and then forwards the output of that XQuery to views/titles-to-html.xql, the output of which is finally rendered in the browser window as the output of the request. This is URL rewriting: the user doesn’t see any changes to the browser address bar even as the database parses the original request to create two new URLs (one for the model and one for the view) to access and execute code.

Let’s test it out and see what we get at this URL: http://localhost:8080/exist/apps/hoaXed/titles

You should see a blank webpage, as shown in the left side of the image above. Depending on your browser, you may see some configuration information, instead of just a blank page, but you won’t see a list of titles. If you see an error message, read it carefully and change one thing at a time as you go back through the directions from the stages. Make sure you saved all changes, your VSCode sync is working, and check all steps.

You want the output to be a list of HTML-tagged titles, and not a blank page, and you can use Developer Tools (available in all browsers) to view the HTML underlying this blank page as a way of learning the source of the incorrect output. The screenshot above shows a single <html:section> that contains a single empty <html:ul>. This isn’t everything you want, but it does tell you a lot of great information. Most importantly, it tells you that 1) the pipeline you created using the controller is functioning, since you see an HTML fragment using elements you wrote in our view XQuery, and 2) you are created a <ul> wrapper for a list of titles, but it unexpectedly contains no list items. You can further confirm that:

• The output of the model module titles.xql is providing titles. You can see that by running it manually in VSCode, bypassing the controller.
• The controller is connecting the model to the view. You know this because you’re seeing the <html:section> and <html:ul> elements that titles-to-html.xql creates.

This means that the problem must lie in the code that transforms the model to the view: although you’ve confirmed that the model contains information about all of the titles, the view doesn’t seem to be able to do anything with that information. Here’s the code you wrote in the titles-to-html.xql file:

xquery version "3.1";
(:==========
Declare namespaces
===========:)
declare namespace hoax = "http://www.obdurodon.org/hoaxed";
declare namespace m = "http://www.obdurodon.org/model";
declare namespace tei = "http://www.tei-c.org/ns/1.0";
declare namespace html="http://www.w3.org/1999/xhtml";

(:=====
the function request:get-data(); is an eXist-specific XQuery
function that we use to pass data among XQuery scripts via 
the controller.
=====:)
declare variable $data as document-node() := request:get-data();

(:=====
HTML rendering begins here
=====:)
<html:section>
    <html:ul>{ 
        for $item in $data/m:item
        return
            <html:li>{$item/m:title || ", " || $item/m:date}</html:li>
    }</html:ul>
</html:section>

You copied the FLWOR loop directly from the tmp file you were using before, where you passed one element into another element using a variable. In your temporary version $data was an element of type <m:list>, but in titles-to-html.xql you declare the type of the $data variable as document-node() rather than as an element. By changing the datatype of the variable from an element to a document node, you introduced an extra path step that is making your XPath no longer correct. While this kind of mistake may be difficult to detect, it is easy to fix: you can either update your XPath to add the additional step or you can change the type of $data to pass and receive an element instead of a document node. You’re going to change the XPath because it’s best to keep these outputs as documents rather than elements or fragments (for API reasons you’ll learn about later).

To fix the code, then, in the line that currently reads:

for $item in $data/m:item

we change / to // to produce:

for $item in $data//m:item`

The original / said to look only at children of whatever $data is; the new // says to look at all descendants of $data, all the way down the document tree, to the expression will find the list items no matter where they’re located in the document. Once you’ve saved this change, when you try accessing http://localhost:8080/exist/apps/hoaXed/titles again you should see something like:

Depending on your choice of browser you may seem some extraneous information at the bottom, below the list, which you can ignore for now. If you’re not seeing the list formatted this way, take some time to troubleshoot before you fork this branch.

Challenge questions

If you were successful, try changing your URL a bit. Each forward slash is a path step within the database, so hoaXed/titles prompts the controller to generate this page, but what would hoaXed/modules/titles.xql show? Why?(Hint: What part of the controller will process this request?) What will hoaXed/views/titles-to-html.xql show you? Why?

Summary

This stage sets up the controller, a key component of the application that allows users to interact with the database via the web browser. Now that you’ve written this code, you can create additional pages and features within the web application in a modular way. In the next stage you’ll create an index (main) page, sometimes called a home page or splash page, and direct users to it by default.

You can find the complete code for this stage at https://github.com/Pittsburgh-NEH-Institute/hoaXed/tree/04-add-controller.