04 add controller
In the previous stage we learned about the model-view-controller (MVC) approach to application architecture. MVC separates the information content (the model) from the way it is presented (as HTML). This allows us to transform, filter, and make calculations with the data when we construct the model, then transform the model into a presentation view. To achieve this, we already started using a namespace for the model, which we bind to the m: namespace prefix.
Namespaces exist to help us differentiate types information. We might have the same element name in our TEI data for a paragraph (<tei:p>) as we do in our model (<m:p>), and the HTML element happens to have the same local element name, too (<html:p>). Why is it important for these elements to be differentiated? They all have the same content and they represent the same type of information regardless of which namespace—but we treat them differently in different contexts. Where appropriate, you can (and we will) transform directly from TEI to HTML without using the model namespace at all, but what if we wanted to provide only the first line of the first paragraph of each article as an incipit on the search results page? It would be inefficient for us to fetch the whole article (or even the whole first paragraph) for this use when what we really need is just the title, the identifier (@xml:id value), and a few words. We could address the incipit requirement by building the model to represent just those first few words in an element called <m:incipit>. If we want to provide the whole paragraph when the user hovers over the incipit, we could use <m:p>. It isn’t surprising that some features of the model will correspond directly to features of the TEI sources, but the source documents and the model are nonetheless not the same, and putting them in different namespaces helps us customize the markup for each of the two purposes.
For most of this laboratory edition’s application features, then, you’ll want to write XQuery that transforms the TEI sources to an appropriate selective, purpose-specific data model, which you’ll then transform to HTML. There are other transformations your edition might require, though. For example, you might want your XQuery to output not only HTML for reading, but also comma-separated values (CSV) you can use in Gephi for network analysis, plus SVG for graphic visualization, plus other formats for other purposes. The data from your TEI sources that supplies the information behind those outputs might be partly (or even entirely) the same, and in that context MVC architecture would let you separately selecting and organizing data for the output from formatting it for presentation. Visitors to your web application won’t know (or care) about whether the output is created all at once or in pipelines, but you, as the developer, will be able to focus on the subcomponents of the larger task individually, which can make for easier planning, implementation, and testing. In MVC architecture the C stands for controller, which is an XQuery resource that receives the user request and specifies how it should be interpreted to create the model and then transform the model into the view. In this session you’re going to 1) create a views subdirectory to hold the XQuery that will transform the model to HTML, 2) create a controller XQuery resource that will ensure that the model and view can find each other, and 3) confirm that the new modular implementation produces the desired HTML output.
In the previous stage, you wrote code that addressed TEI XML to generate the model, and then you assigned that to a variable in order to pass it through to code that could transform it to HTML. We want to separate these steps from each other, but in order for that to happen, we have to write a controller.
The controller is a file the database uses to 1) receive requests for information (when, for example, the user clicks on a link), 2) invoke the XQuery that creates the appropriate model, and then 3) pass that model through one or more XQuery transformations to create and return the view as HTML. Inputs from the user come from the web browser, either via a direct URL they type, or by clicks that provide that URL. If we want to make the XQuery transformations we write accessible to a user who doesn’t know what XQuery is, or if we want to create an interactive reading view for research (or any number of other goals that involve providing input to the database), our controller must include directions for handling these inputs.
We start this stage by returning to the workstation, reconnecting with the database, and practicing our workflow. Return to the first section below throughout the stages for a reminder of how to get set up.
Next, we move the code we wrote in Stage 03 into their own files, in their own directories. Then we add a new file, called controller.xql to the main project directory and paste in some code. While we will not go into depth about why and how the controller works, you can reference eXist-db page 194, “URL mapping using URL rewriting” for more detailed explanations.
Once we have the controller in place, we begin troubleshooting. [TROUBLESHOOTING GUIDE]
- Start the eXist database. The easiest way to do this is to open the application on your desktop. While you will see some waiting-related screens, it won’t open a graphical user interface (GUI) like you would expect from something like a text editor. Instead, you can confirm it is running by checking the widget it creates (either top or bottom tray, depending on your operating system), or by visiting http://localhost:8080.
- Start VSCode by opening the application. Open your project's workspace. Once you confirm the database is running and you're using the workspace, start synchronization using the command palette, or the auto sync button in the lower right hand corner of the editor.
- Once you confirm your connection, open a shell and navigate to the main directory of your project. Use
mkdir viewsto create a new directory. - Use
cd viewsto navigate into this directory. From there, you can create a new XQuery file (use .xq or .xql filetype). We use the filename titles-to-html.xql and will continue using that naming convention. - Next, we want to add namespaces, so we copy them from titles.xql.
xquery version "3.1";
(:==========
Declare namespaces
===========:)
declare namespace hoax = "http://www.obdurodon.org/hoaxed";
declare namespace m = "http://www.obdurodon.org/model";
declare namespace tei = "http://www.tei-c.org/ns/1.0";
declare namespace html="http://www.w3.org/1999/xhtml";
- Now we'll add the XQuery we wrote to process the output of titles.xql from the previous stage. You can find this in the 03-tmp-HTML branch in the file titles-html.xql.
<html:section>
<html:ul>{
for $item in $data/m:item
return
<html:li>{$item/m:title || ", " || $item/m:date}</html:li>
}</html:ul>
</html:section>
So far, all we’ve done is move code from one file to another. In the previous stage, the for loop that creates our <html:li> items used a variable called $data that we declared in the same file. Since we aren’t declaring $data yet in this context, we see a warning variable $data is not set. Let's set the variable by adding the following after our namespaces.
declare variable $data as document-node() := request:get-data();
The function request:get-data() returns the content of a POST request, which is a name for a specific type of API call. We will talk about APIs shortly, but in this tutorial suffice to say that we need this content to output from our model titles.xql file, and we want it to be a document node (rather than an XML element or a string, for example). In order for this to work, we need to write a controller file that will connect the model to the view and the view to the user.
Here’s the controller in full. We will write this to the main project repository and save the file as controller.xql. It is also available in the repository from the Stage 04 branch onwards.
xquery version "3.1";
declare namespace html="http://www.w3.org/1999/xhtml";
declare variable $exist:root external;
declare variable $exist:prefix external;
declare variable $exist:controller external;
declare variable $exist:path external;
declare variable $exist:resource external;
declare variable $uri as xs:anyURI := request:get-uri();
declare variable $context as xs:string := request:get-context-path();
declare variable $ftcontroller as xs:string := concat($context, $exist:prefix, $exist:controller, '/');
if ($exist:resource eq '') then
<dispatch xmlns="http://exist.sourceforge.net/NS/exist">
<redirect url="index"/>
</dispatch>
else
if (not(contains($exist:resource, '.')))
then
<dispatch xmlns="http://exist.sourceforge.net/NS/exist">
<forward url="{concat($exist:controller, '/modules', $exist:path, '.xql')}">
<add-parameter name="exist:root" value="{$exist:root}"/>
<add-parameter name="exist:controller" value="{$exist:controller}"/>
<add-parameter name="exist:prefix" value="{$exist:prefix}"/>
</forward>
<view>
(:transformation to html is different for different modules:)
<forward url="{concat($exist:controller, '/views/', $exist:path, '-to-html.xql')}"/>
</view>
<cache-control cache="no"/>
</dispatch>
else
<ignore xmlns="http://exist.sourceforge.net/NS/exist">
<cache-control cache="yes"/>
</ignore>
Now that we have this file saved, let’s break it down in human language. We declare some variables, some of which are included by default in our eXist-db instance. Others are strings or URIs we fetch or construct using functions. Next, we have an if/then/else statement that evaluates some input and provides some output. If there’s no input ($exist:resource eq ''), the controller will redirect the user to index, which we will add to the app at a later stage.
If there is an input and it doesn't contain a dot ((if (not(contains($exist:resource, '.')))), then we forward the user to a URL we constructed using the input from the URL the user gave us. This is that POST request we were talking about earlier. The new URL we construct is {our app's base URL}/modules/{user input}.xql. This means that if we go to our browser and type the app's base URL plus ‘/titles’, our controller can access and execute the titles.xql file we just wrote. Then, it can forward the output of that to the correct file in the ‘views/’ directory by constructing the URL for that as well. All of this happens without the browser changing, so the user should just see a new page. This is URL rewriting: the user doesn’t see any changes to their URL bar, even as the database parses it to create new URLs to access and execute code.
Let's test it out and see what we get at this URL: http://localhost:8080/exist/apps/hoaXed/titles
You should see a blank webpage. If you see an error, read the error carefully and change one thing at a time as you go back through the directions from the stages. Make sure you saved all changes, your VSCode sync is working, and check all steps [TROUBLESHOOTING GUIDE].
If you see a blank webpage, you can use Developer Tools available in most browsers to view the HTML underlying this blank page. As you see in the screenshot above, we have an html:section that contains one empty html:ul. This isn’t what we wanted, but it does tell us a lot of great information.
First, we are successfully executing all the code we wrote without errors. It may not be working as we wanted, but the pipeline we created using the controller is functioning, because we have an HTML fragment using elements we wrote in our view module. We can confirm the output of the model module titles.xql is providing correct output by running it manually in VSCode. With this information, where is there likely to be an issue?
We should check the view. We can tell information is being passed somewhat correctly because the html:ul element is generated. We are missing the html:li elements completely. Here's the code we wrote in the titles-to-html.xql file.
xquery version "3.1";
(:==========
Declare namespaces
===========:)
declare namespace hoax = "http://www.obdurodon.org/hoaxed";
declare namespace m = "http://www.obdurodon.org/model";
declare namespace tei = "http://www.tei-c.org/ns/1.0";
declare namespace html="http://www.w3.org/1999/xhtml";
(:=====
the function request:get-data(); is an eXist-specific XQuery
function that we use to pass data among XQuery scripts via
the controller.
=====:)
declare variable $data as document-node() := request:get-data();
(:=====
HTML rendering begins here
=====:)
<html:section>
<html:ul>{
for $item in $data/m:item
return
<html:li>{$item/m:title || ", " || $item/m:date}</html:li>
}</html:ul>
</html:section>
We copied the FLWOR loop directly from the tmp file we were using before, where we passed one element into another element using a variable. If you notice, now the $data variable declares this as document-node() rather than an element. By changing this from an element to the document node, we introduced an extra path step that is making our XPath no longer correct. While this kind of mistake is difficult to detect, it is easy to fix. We can either update our XPath, or we can update our variable to pass an element instead of a document node. We decided to change the XPath, because we would prefer to keep these outputs as documents rather than elements or fragments (for API reasons we will learn about later down the road).
So, in the for $item in $data/m:item line, we add an extra path step to make it for $item in $data//m:item, skipping down the tree to the item elements over which we want to loop. Now that we have saved this change, we can try accessing (http://localhost:8080/exist/apps/hoaXed/titles)[http://localhost:8080/exist/apps/hoaXed/titles] again.
Your browser should look something like this. If you’re not seeing the list formatted this way, take some time to troubleshoot before you fork this branch. If you were successful, try changing your URL a bit. Each forward slash is a path step within the database, so hoaXed/titles prompts the controller to generate this page, but what would hoaXed/modules/titles.xql show? Why? What will hoaXed/views/titles-to-html.xql show you?
This stage sets up the controller, a key component of our application that allows users to interact with the database via the web browser. Now that we’ve written this code, we can create many new pages and features within the web application in a modular way. In the next stage, we add an index page, sometimes called a home page and direct users to it by default.
You can find the complete code for this stage at https://github.com/Pittsburgh-NEH-Institute/hoaXed/tree/04-add-controller.