Server-Side Javascript Files as Pages with HTML5 pushState ~ 2015-05-31

Suppose I have a website that needs to behave as both a single-page site and a multi-page site. For example, let's say I want the top-level index page of my website to build a stage and load some data that can be displayed in arbitrarily many ways. I can build links and buttons to manipulate the data through Javascript, but my URL never changes and I can't easily return to a particular view without retracing those clicks. Each click could load a whole new page, but that would require redrawing the stage and reloading all of the data. For a large dataset this would make for a halting and unpleasant user experience. Fortunately HTML5 provides a solution.

For a trivial example to demonstrate the concept here's a demo:

...

Click any of the links on the left and you should see the field on the right be populated with new values. This alone is nothing fancy, but take note of the the URL of this page in your browser as you click the links - it should be changing along with your clicks. You should also be able to use your browser's back and forward buttons to retrace your steps, or you can reload this page with a search query to go directly to one of the links.

This is achieved by building separate Javascript files for each link (or page), loading them with some DOM injection, and manipulating the browser history using HTML5's history object (specifically the pushState() method). Read on to see how it's done!

Javascript Files as Pages

To get started we need a pattern for each page. Javascript files are a natural fit here because if each page is simply presenting existing data loaded in client memory in a different way then we're going to do that with Javascript anyway. Let's start with a Page class:

var Page = function(){
    this.id = null;     // URL-friendly string
    this.title = null;  // Arbitrary string
    this.is404 = false; // Boolean
    this.load = null;   // Function to contain all page logic
};

// Method to convert arbitrary title string into a URL-friendly ID string
Page.prototype.titleToId = function(){
    return this.title.toLowerCase().replace(/\W/g,"_");
};

// Method to set title and automatically set ID
Page.prototype.setTitle = function(title){
    this.title = title.toString();
    this.id = this.titleToId();
};

Pages are deliberately simple objects. Each has an ID and a title, both of which will be necessary for browser history. The titleToID method converts any string to just lower case alphanumerics with underscores to stand in for all other characters. For example, Javascript Files as Pages becomes javascript_files_as_pages. The ID will be our primary unique identifier and will appear in URLs when we get browser history working.

Tracking whether a page is the catch-all error page with the is404 boolean will come in handy as we build in fault tolerance later. Finally, the load paramater is an arbitrary function where all the page's logic should live.

Before we can build our first page, though, we need the navigation scaffolding.

Navigation Scaffolding

Let's start with another class for navigation. Inside this class we'll want to keep a copy of the page we're currently on and the next page we've just loaded. As will become apparent a bit later it will also be useful to track to the ID of the page we've attempted to load separately. Finally we'll also want to keep a cache of all loaded pages for quickly reloading ones we've already seen upon request.

var Nav = function(){
    this.current = null; // Page
    this.next = null;    // Page
    this.attempt = null; // Page.id
    this.cache = {};     // associative array: Page.id => Page
};

Next we're going to want a method on our Nav object to call up a new page by its ID:

Nav.prototype.call = function(page_id){
    this.next = null;
    (function(nav, page_id){
        nav.fetch(page_id, function(){ // Fetch the next page from the server
            nav.next.load(function(){  // Execute the next page's load function
                nav.finalize();        // Page loaded: complete the transaction
            });
        });
    })(this, page_id);
};

There's a lot going on here. The first thing the call method does is null out the next attribute on the Nav object to make way for the page we're going to load. We then have a cascade of three functions that fire inside a closure: nav.fetch(), nav.next.load(), and nav.finalize(). These functions fire as chained callbacks because, since we're talking to a server to get the new page, we simply don't know when each action will complete. By wrapping the entire cascade in a closure and passing in this as the variable nav we can ensure the nav object remains in scope for each function whenever it needs to ultimately fire.

So what do those other Nav methods look like?

Fetching Javascript with Javascript

Loading a Javascript file from the server into a page that's already been loaded (with Javascript) is not the most straightforward task, but it's not impossible. That's what the fetch() method does, and here's how it looks:

Nav.prototype.fetch = function(page_id, callback){
    
    // If the page is already in the cache then fetch it from there
    if (typeof this.cache[page_id] != "undefined"){
        this.next = this.cache[page_id];
        callback();
        
    // Otherwise we need to load it from the server
    } else {
        
        // Store the ID we're attempting to load
        if (page_id != '404'){ this.attempt = page_id; }

        // Execute the load in a closure
        (function(nav, page_id, callback){
            
            // Compose the page URI
            var src = page_id + '.js?t=' + new Date().getTime();

            // Create a new script element in the document's head
            var head = document.getElementsByTagName("head")[0] || document.documentElement;
            var script = document.createElement('script');
            script.type = 'text/javascript';

            // Set the new script element's src, onload, and onerror values
            script.src = src;
            script.onload = function(){
                nav.next = nav.cache[page_id];
                callback();
            }
            script.onerror = function(){
                nav.next = nav.cache["404"];
                callback();
            }

            // Append the new script element to the page to trigger the loading
            head.appendChild(script);
            
        })(this, page_id, callback);
    }
};

The first thing to notice is our built-in caching. By storing pages in the Nav.cache associative array by ID we can easily skip the cumbersome step of loading an external Javascript file if we've already done that.

The next thing to notice would be our 404/error handling. To use this scaffolding gracefully it's important to create a special "404" page (with an ID and title of "404") so that we can serve up something in case a page is called which doesn't exist.

Finally there's the big closure that does all the magic. Essentially, to load a Javascript file on a page that's already completely rendered, we need to inject a new <script> tag. Doing this with Javascript as we are we can set its source appropriately as well as trigger callbacks when it loads or errors. Those onload and onerror methods are why this entire block is in a closure, too: those methods need access to our callback method and our global nav object, and remember that closures create a persistent scope for all subsequent methods within. It's also important to note here how we wait until we've completely defined our new script element before we inject it into the DOM with head.appendChild(script);.

Post-load Actions and Browser History

So now we've called our page by its ID and fetched the Javascript source for it. That source was injected into the DOM as a <script> element so that it executed immediately, and that triggered the firing of Nav.next.load() - an arbitrary function specific to the page we've loaded. That's now triggered the callback to our last function in the chain: Nav.finalize().

Nav.prototype.finalize = function(){
    this.current = this.next;
    this.next = null;
    this.pushUrl();
};

Ah, something simple! Basically we bump the current page and replace it with the next page, the next page becomes null, and we then trigger a call to pushUrl(). This is the method that handles our browser history.

// Push URL and history for enabled browsers
Nav.prototype.pushUrl = function(){
    if (Modernizr.history){ // Use Modernizr to make sure the client's browser has history methods
        var id = this.current.id;
        var title = this.current.title;
        if (this.current.is404){
            id = this.attempt;
            title = '404 - Page Not Found';
        }
        var state = { page: id };
        var uri = "?" + id;
        history.pushState(state, title, uri);
    }
};

The first thing to note here is that we're useing Modernizr to detect if the client has the history methods we need. Modernizr is nice lightweight Javascript library for detecting all sorts of features. It's inclusion in projects is essential to maintain a wide range of browser compatibility when using features with less-than-universal browser support. It's also very easy to use as it creates an object with a slew of booleans you can just check. In this case our pushUrl() won't work unless a user has HTML5 history controls, and Modernizr will give me that boolean if I load it with my original page load.

Now then, for capable browsers what we're doing here is first gathering info about where we are from the current page in the Nav object. Here's where the attempt parameter becomes useful. If the attempt failed and we're on a 404 page we can still pass the failed ID into the history instead of the "404" id. Finally we bundle it all together an pass it to history.pushState().

This is where all the magic happens. With just that one method we've not only updated the browser history to make it look like we're on a new page (such that our back button will take us to the previous page) but our browser's URL will reflect the new page we're on. For example, if our home page is http://foo.bar and we've loaded the page baz via this method our URL will now show http://foo.bar?baz with a new accompanying browser history entry even though we never reloaded the entire page.

Putting it all Together

Now to actually put this into action we need to define instances of our classes. Let's start with an example of defining an instance of the Nav class in the global scope:

var nav = new Nav();
nav.cache['default'] = { title: "Default",
                         id: "default",
                         is404: false,
                         load: function(callback){
                             // Load the default page
                             callback();
                         }
                       };
nav.cache['404'] = { title: "404 - Page Not Found",
                     id: "404",
                     is404: true,
                     load: function(callback){
                         // Load the 404 page
                         callback();
                     }
                   };
nav.call("default");

Defining a global Nav object is very straightforward. It's important to also define common pages like the 404 page and perhaps the default/initial page such that they don't need to be loaded later through script element injection via the Nav.fetch() method.

/* Page: hello_world.js */
var page = new Page();
page.setTitle("Hello World");
page.load = function(callback){
    // Page-specific logic goes here
    callback();
}
nav.cache[page.id] = page;

Note that the load function needs to accept a single parameter for a callback function which it calls at the end of its logic. This pattern is how the scaffolding knows the loading function is complete so that it can finalize by manipulating the history object.

We can now call this page using the scaffolding. This could be triggered by clicking a link or other navigation element, or really any conceivable event:

nav.call("hello_world");

Working from the URL Backwards

While all this will get navigation up and running from the home page we're missing one final crucial element: returning to a given page directly from a URL. Let's say I like the Hello World page so much that I bookmark it and want to return to it from the URL: http://foo.bar?hello_world. Since the URL still calls the same top-level index page for the website where I've included all my scaffolding files, Modernizr, and defined my global Nav object all I need to do is include a check at load-time to see if the URL includes a request for a specific page:

if (window.location.search.slice(1).length){
    var id = window.location.search.slice(1);
    nav.call(id);
} else {
    nav.call("default");
}

The DOM stores the "search" parameter of a URL (everything after the first question mark) in window.location.search. Since this value includes the question mark itself we need to slice the string to remove the first character. We can then feed that value directly into nav.call() without fear. Because we took the time to include graceful 404 handling any visitor trying weird search parameters that don't map to valid pages will just get a 404 through the fetch method instead of seeing things break. And, for those users who liked our content and want to get back to it without all the clicks, every page is now fully bookmarkable and can serve as an entry point to the site.

Last Step: Detecting popstate Events

The final step in making this all work end-to-end is detecting what's called a popstate event. The term popstate has the same etymology as pushState in that the history object behaves like a stack. I don't know why the former isn't camel-cased and the latter is but either way a popstate event is triggered whenever a browser's back or forward buttons are clicked. We can define an event listener for these events to essentially tie those browser buttons to our nav.call() method when the history is telling us to load a specific page.

// Listen for popstate events (users navigating history through
// browser controls) to make that navigation work as expected
window.onpopstate = function(event) {
    if (typeof event.state == "object"){
        if (typeof event.state.page == "string"){
            nav.call(event.state.page);
        }
    }
};

Since we don't want to completely overtake the user's back and forward buttons when they get into history before or after our website our event listener first makes sure there's a state object in the event and then makes sure that the state object has a page attribute. Recall in our Nav.prototype.pushUrl() method we created a state object with a page attribute set to the ID of the page for the history entry. As the popstate event triggers this same object is what we're capturing and feeding back into our nav.call() method, so we're safe if the popstate event returns history that we didn't create (in such a case the default back/forward action will happen).

The Real Last Step: Special-Casing popstate Events

It sure looks like we're done, but not quite. When we trigger a nav.call() on a popstate event that call will cascade into nav.finalize() and, subsequently, nav.pushUrl(). That method will then write the current state to the history even though we've just pulled it from history (say, with a back button). In practice this approach of firing pushState every time we finalize will make our back or forward buttons only work once.

To get around this we just need to pass some context so that whenever we're calling a new page from a popstate event we don't fire nav.pushUrl(). First we can modify the scaffolding to pass through an additional optional argument:

Nav.prototype.call = function(page_id, frompopstate){
    this.next = null;
    if (typeof frompopstate == 'undefined'){ var frompopstate = false; }
    (function(nav, page_id, frompopstate){
        nav.fetch(page_id, function(){      // Fetch the next page from the server
            nav.next.load(function(){       // Execute the next page's load function
                nav.finalize(frompopstate); // Page loaded: complete the transaction
            });
        });
    })(this, page_id, frompopstate);
};

Nav.prototype.finalize = function(frompopstate){
    this.current = this.next;
    this.next = null;
    if (!frompopstate){
        this.pushUrl();
    }
};

And then we can modify just the nav.call() instance in the onpopstate event handler to pass the flag:

// Listen for popstate events (users navigating history through
// browser controls) to make that navigation work as expected
window.onpopstate = function(event) {
    if (typeof event.state == "object"){
        if (typeof event.state.page == "string"){
            nav.call(event.state.page, true);
        }
    }
};

And now pushUrl(), which triggers the pushState, will only fire when appropriate. Our browser's back and forward buttons should now work 100% as expected.

**

And that's it! With this basic scaffolding it's possible to build an entire functioning website with pure dynamic loading of javascript files. This will effectively provide the ability to have arbitrarily many pages loadable and navigable within a single page that never reloads.

A Real-World Example

To see another real-world use case of this, take a look at Nuclides.org. The site all loads from a top-level index file and is built as a platform for arbtirarily many "questions" (pages) about the elements and their isotopes. The data set is loaded once and the default/home page of What is Nuclides.org? can be accessed from the top-level URL or from the URL with the page ID loaded up.

Each page's load function injects new text content onto the stage while rearranging the presentation of the data. For example, What is a Nuclide? shows the entirety of the isotope/nuclide data set while How do Atoms Decay? shows only one nuclide from the data set at a time along with an interactive dashboard for playing with atomic decay modes. The modular approach to pages this way allows complex logic, like the decay dashboard in the latter example, to be kept separate from the main application logic in its own page.

This concept can be extended further, too, by using other elements of the URL. For example, in addition to window.location.search there is window.location.hash, everything that appears after the octothorpe (#). Nuclides.org utilizes this to form a pattern where a single page can be used for any of the 118 elements, as seen on the What is Hydrogen? page. Only one Javascript file exists for the element pages (with an ID of what_is_element) and the URL appends the element name after the page ID with the hash (#) mark. This does require special handling throughout the scaffolding but it's an effective way at turning this system of "pages" into a system of "pages" and "subpages", as it were.

Full Scaffolding Code

Recall there are are two pieces to the scaffolding code: the class definitions themselves and then the instance and event handler definitions.

Class Definitions

var Page = function(){
    this.id = null;     // URL-friendly string
    this.title = null;  // Arbitrary string
    this.is404 = false; // Boolean
    this.load = null;   // Function to contain all page logic
};

// Method to convert arbitrary title string into a URL-friendly ID string
Page.prototype.titleToId = function(){
    return this.title.toLowerCase().replace(/\W/g,"_");
};

// Method to set title and automatically set ID
Page.prototype.setTitle = function(title){
    this.title = title.toString();
    this.id = this.titleToId();
};

var Nav = function(){
    this.current = null; // Page
    this.next = null;    // Page
    this.attempt = null; // Page.id
    this.cache = {};     // associative array: Page.id => Page
};

Nav.prototype.call = function(page_id, frompopstate){
    this.next = null;
    if (typeof frompopstate == 'undefined'){ var frompopstate = false; }
    (function(nav, page_id, frompopstate){
        nav.fetch(page_id, function(){      // Fetch the next page from the server
            nav.next.load(function(){       // Execute the next page's load function
                nav.finalize(frompopstate); // Page loaded: complete the transaction
            });
        });
    })(this, page_id, frompopstate);
};

Nav.prototype.fetch = function(page_id, callback){
    
    // If the page is already in the cache then fetch it from there
    if (typeof this.cache[page_id] != "undefined"){
        this.next = this.cache[page_id];
        callback();
        
    // Otherwise we need to load it from the server
    } else {
        
        // Store the ID we're attempting to load
        if (page_id != '404'){ this.attempt = page_id; }

        // Execute the load in a closure
        (function(nav, page_id, callback){
            
            // Compose the page URI
            var src = 'js/' + page_id + '.js?t=' + new Date().getTime();

            // Create a new script element in the document's head
            var head = document.getElementsByTagName("head")[0] || document.documentElement;
            var script = document.createElement('script');
            script.type = 'text/javascript';

            // Set the new script element's src, onload, and onerror values
            script.src = src;
            script.onload = function(){
                nav.next = nav.cache[page_id];
                callback();
            }
            script.onerror = function(){
                nav.next = nav.cache["404"];
                callback();
            }

            // Append the new script element to the page to trigger the loading
            head.appendChild(script);
            
        })(this, page_id, callback);
    }
};

Nav.prototype.finalize = function(frompopstate){
    this.current = this.next;
    this.next = null;
    if (!frompopstate){
        this.pushUrl();
    }
};

// Push URL and history for enabled browsers
Nav.prototype.pushUrl = function(){
    if (Modernizr.history){
        var id = this.current.id;
        var title = this.current.title;
        if (this.current.is404){
            id = this.attempt;
            title = '404 - Page Not Found';
        }
        var state = { page: id };
        var uri = "?" + id;
        history.pushState(state, title, uri);
    }
};

Instance and Event Handler Definitions

// Define the nav object with default and 404 pages
var nav = new Nav();
nav.cache['default'] = { title: "Default",
                         id: "default",
                         is404: false,
                         load: function(callback){
                             document.getElementById("page_content").innerHTML = "This is the default page.\nWelcome to the pushState demo!";
                             callback();
                         }
                       };
nav.cache['404'] = { title: "404 - Page Not Found",
                     id: "404",
                     is404: true,
                     load: function(callback){
                         document.getElementById("page_content").innerHTML = "404 - Page Not Found\nAttempted page id: " + nav.attempt;
                         callback();
                     }
                   };

// Load the first page (from serach query if provided or just the default)
if (window.location.search.slice(1).length){
    var id = window.location.search.slice(1);
    nav.call(id);
} else {
    nav.call("default");
}

// Listen for popstate events (users navigating history through
// browser controls) to make that navigation work as expected
window.onpopstate = function(event) {
    if (typeof event.state == "object"){
        if (typeof event.state.page == "string"){
            nav.call(event.state.page, true);
        }
    }
};

And, just for good measure, here's the source for the Hello World and Fibonacci pages:

/* Page: hello_world.js */
var page = new Page();
page.setTitle("Hello World");
page.load = function(callback){
    document.getElementById("page_content").innerHTML = "This page just prints a message.\nHello World!";
    callback();
}
nav.cache[page.id] = page;

/* Page: fibonacci.js */
var page = new Page();
page.setTitle("Fibonacci");
page.load = function(callback){
    var seq = [ 1, 1 ];
    var n = 0;
    while (n < 10){
        seq.push(seq[seq.length-1]+seq[seq.length-2]);
        n++;        
    }
    document.getElementById("page_content").innerHTML = "This page generates the Fibonacci Sequence.\n[" + seq.toString() + "]";
    callback();
}
nav.cache[page.id] = page;