Yahoo! User Interface Blog (YUIBlog)

Dissecting 3.5.0 DataTable, Part 1

DataTable has been one of YUI's most heavily used and relied upon widgets for years. In 3.5.0, DataTable got a major overhaul, resulting in some small changes to the API and some big changes to the infrastructure.

In this two part article, we'll talk about these changes, what led up to the decision to revisit the infrastructure rather than focus on features, break down some of the new and experimental concepts being explored, and finish up with a look at the plan for the future of DataTable as we see it today.

So without further ado, let's address the obvious question:

Why Refactor?

The short answer is "to make DataTable easier to use and customize". The 3.4.1 infrastructure was based on a "core" Widget model, with all additional features left to plugins or subclasses. The data was held in a Y.Recordset instance and the column configuration in a Y.Columnset instance. The Widget was responsible for creating the entire table markup. That seemed like a reasonable architecture, but there were a few issues with this:

1. DataTables are defined by their features, which means there could be a lot of plug()ing to get the DataTable that fits your application. This made working with DataTable too complicated and verbose.
2. The contract of the Plugin API hindered DataTable's multiple features from interacting properly. In fact, certain DataTable features in 3.4.1 had already started breaking this contract.
3. DataTable was hard coded to use Y.Record, Y.Recordset, Y.Column, and Y.Columnset. This coupling limited the ability to customize DataTable or integrate it with other components in your application.
4. The rendering algorithm was flexible, but its hooks for customization were fixed. There was an opportunity to make rendering more customizable (and a whole lot faster) without requiring implementers to do major surgery on their instances.

The new architecture aims to address these and other issues uncovered during its production life prior to 3.5.0, and take advantage of the App Framework components that weren't around when DataTable was initially created in 3.3.0.

DataTable is incredibly important to a lot of people, and it's important to us that it not only be built right, but also that it be as easy to use as possible. These changes aim to move DataTable in the right direction.

Big Changes

The major changes from 3.4.1 to 3.5.0 are mostly under the hood, and with any luck, migrating to 3.5.0 DataTable should be painless.

Most of the outward changes to the API make it much easier to add and use DataTable features. Compare a 3.4.1 sortable DataTable to its 3.5.0 equivalent:

What's in a Name?

The first thing you'll notice is that as of 3.5.0, you create instances of Y.DataTable rather than Y.DataTable.Base. The base class is still around, but now its primary role is as a superclass for extension (which we'll talk more about in the next article).

Y.DataTable is now the main class and the namespace for classes that support DataTable, such as Y.DataTable.Base, Y.DataTable.Sortable, and Y.DataTable.BodyView. The notable exception is Y.Plugin.DataTableDataSource, which hasn't yet been migrated over to the new world order (expect that in 3.6.0).

You'll see more about this in the next article when we talk about some of DataTable's new concepts.

Features as Class Extensions

The 3.5.0 architecture marks a shift away from the plugin model for DataTable. Instead, we're putting features in class extensions, and using the powerful Y.Base.mix() method to augment these features into the DataTable class, creating a sort of ad-hoc multiple inheritance model.

The two primary benefits this approach has are:

1. The API for your DataTable and all its features is in one place, your Y.DataTable instance.
2. Rather than having to explicitly build up your DataTable from the Base, simply including the feature modules in your use() line adds support for those features to the Y.DataTable class.

But what if you have a couple of DataTables on the page, and you don't want all of them sortable or scrollable, etc.?

To keep features from leaking between DataTable instances, each feature includes a behavioral trigger, which is responsible for activating the feature for that particular instance. A feature might look for assigned table attributes, such as "scrollable", or column configurations such as "width". It might even have multiple triggers, such as datatable-sort looking for either the "sortable" table attribute or "sortable" column configuration property.

No More Coupling

As of 3.5.0, Y.Columnset and Y.Column are gone, and you're not limited to data storage in Y.Record and Y.Recordset.

Column definitions are now stored as a simple array of objects, just like what you pass into the DataTable's columns attribute.

The reason for this change is that many features are column-specific, making the column definitions a natural place to configure them. But the Column class shouldn't include feature related attributes unless that feature is requested. However, features implemented as plugins should not directly modify classes. So how can a column definition include sortable: true if the Column class doesn't have a sortable attribute and the plugin can't add it? There wasn't a good answer, so we (regrettably) added the feature attributes to the base Column class, allowing them to do nothing unless the plugin was added. With so many pending DataTable features wanting to be configured in the column definitions, we decided to remove the Column class wrappers in 3.5.0 to allow columns to be defined with whatever properties were needed. The jury is still out on this change, so we're interested in your feedback.

On the data side of things, Recordset and Record were technically working fine, but with the advent of Y.Model and Y.ModelList in 3.4.0, there was now a huge opportunity to share classes that were likely to be used elsewhere in the application. This meant, however, that DataTable would need to be decoupled from its data storage classes because the common use case for the App Framework components is to create subclasses of them with attributes and APIs that specifically encapsulate your business logic.

So in 3.5.0, rather than being bound to Y.Record, data records are stored in Model subclass instances. By default, DataTable will create a Model subclass for you, customized to your data, but you now have the option to specify the class to use by setting the DataTable's recordType attribute. Similarly, you can provide your table's data in a ModelList or ModelList-like object.

Now if you share your ModelList or Models across various parts of your application, modifications to any of them will automatically update the table.

Customizable Rendering

In 3.5.0, the rendering algorithm was completely rewritten, and is now significantly faster and more configurable than its 3.4.1 counterpart. The migration guide and user guide go into detail about the various cell formatting options, but one important change worth talking about here is that the entire algorithm can be replaced with a simple configuration change.

In keeping with the move to configurable data storage classes, DataTable now delegates its rendering algorithms to Y.View classes. In 3.5.0, the basic, featureless DataTable has attributes headerView, bodyView, and footerView. By default, footerView is unset, but headerView is set to Y.DataTable.HeaderView, and bodyView is set to Y.DataTable.BodyView.

When a DataTable instance is render()ed, the only DOM content the Widget itself is responsible for is the <table> and its immediate children. The rendering of the header content, footer content, and data rows is left to whatever View class is supplied to the respective attribute.

The default Views provided in 3.5.0 include enough column configuration options to handle most use cases, but if you have special requirements for your table rendering or you want to hyper-optimize your specific implementation, you can supply your own View classes to the attributes noted above, and DataTable will use them instead.

More to Come

The next article will go into more detail about DataTable's architectural techniques, experimental patterns, and what's coming in the next release.

In the mean time, be sure to let us know in the yuilibrary.com forum, on Twitter, and in the #yui IRC channel on freenode, how things are going in your DataTable apps and what features you're most looking forward to.

Find conversations: bloglines | technorati

Share and extend: Bookmark with del.icio.us | digg it! | reddit!

YUI: Open Hours Thurs May 3rd

Fiddling Around With DataTable

To me, it feels like every other Open Hours is about DataTable, but I guess that’s not actually true. So let’s talk about DataTable!

In particular, I want to talk about two things:

1. Tips and tricks with column configurations and cell formatters
2. What’s fast, what’s slow, and how to maximize DT performance

It will be a lot of live coding on jsfiddle, though I’ll try to have a few examples of things prepped. If there’s a particular example you’d like to see, a formatting question you’d like answered, or generally any other DT question or feedback, please bring it.

Recording

The recording is available in the YUILibrary YouTube channel, though the quality didn’t come out very well.

Mentioned

• Flyweight Proxy ModelList
• More precise DOM updates from data changes
• Custom bodyView with simpler rendering logic

Find conversations: bloglines | technorati

Share and extend: Bookmark with del.icio.us | digg it! | reddit!

Generalizing Code Through Functional Programming

Abstraction is a hot buzzword, but many people say abstraction when they really mean generalization. These two concepts are very different. In fact, they apply to opposite ends of the coding process.

A lot has been written about how abstractions simplify the job of constructing software, when programming in the large. Unfortunately, abstractions leak, as demonstrated by Joel Spolsky and Jeff Atwood. This is unsurprising, given the definition of abstraction: the process of considering something independently of its associations, attributes, or concrete accompaniments. In other words, abstractions ignore details, and this always comes back to bite us.

Generalization, on the other hand, is useful when focusing on a specific problem, i.e., programming in the small. The goal of generalization is to make a solution for one problem usable for a larger class of related problems. This is the ultimate in code reuse: build it once and use it over and over again. As long as the generalization is done correctly, there is no danger of it failing, though there is always the possibility of not taking it far enough.

There are many ways to generalize code. This article will focus on building higher order functions, a particular strength of functional languages. Let us start with two obvious examples: map and reduce which apply a function to each element in a collection. These generalize the concepts of generating a new collection or value, respectively, from an existing collection.

It should be possible to generalize map and reduce further, to work on a forest, i.e., a collection of trees, instead of only a collection of items. The problem, however, is that the code that recurses must know how the trees are stored. Is a tree simply nested arrays, e.g., [1, [2, 3], 4], or is each node an object from which the array of child nodes must be extracted? For reduce, one can pass in a function that extracts the children:

function reduceForest(roots, initial, operation, children)
{
	return Y.Array.reduce(roots, initial, function(value, root)
	{
		return reduceForest(children(root), operation(value, root), operation, children);
	});
}

For map, however, it is more complicated, because there is also the question of what the result should look like. A nested array can be mapped to another nested array, and a tree of objects can be mapped to another tree of objects, but supporting both at the same time would make the code quite complicated. Knowing when to stop generalizing is just as important as knowing when to keep going!

We can branch out in a different direction, however, because there are many types of collections, but the above code requires the children to be stored in arrays. We need to generalize the concept of iteration.

YUI’s oop.js provides this, in the private function dispatch. Here is the slightly adjusted version used by gallery-funcprog:

function dispatch(action, o)
{
	var args = Y.Array(arguments, 1, true);
	switch (Y.Array.test(o))
	{
		case 1:
			return Y.Array[action].apply(null, args);
		case 2:
			args[0] = Y.Array(o, 0, true);
			return Y.Array[action].apply(null, args);
		default:
			if (o && o[action] && o !== Y)
			{
				args.shift();
				return o[action].apply(o, args);
			}
			else
			{
				return Y.Object[action].apply(null, args);
			}
	}
}

This is wonderfully general. It works for any action: map, reduce, filter, etc. Arrays are routed to Y.Array. Objects that implement the action are called directly, allowing individual classes to optimize individual actions. All other objects are operated on by the generic versions in Y.Object (provided by gallery-object-extras). Sharp readers will note that the order of iteration for object members is undefined, but this doesn’t matter for map, reduce with commutative operators, filter, etc. (Use Y.some at your own risk, however.)

The option for objects to implement custom versions of the actions leads to yet another generalization: gallery-iterable-extras. This mixin implements all the actions for any object that provides the iterator method. The only requirement is that iterator must return an object that exposes next and atEnd. This is especially efficient for linked lists, where indexed lookup is O(n), but it could also simplify other classes, e.g., Y.NodeList, because one then does not have to explicitly implement map, reduce, filter, etc.

Of course, generalization doesn’t have to be this complicated. When I was writing gallery-sort-extras, I first built this function:

Y.Sort.compareKeyAsString = function(key, a,b)
{
    return compareAsString(a[key], b[key]);
};

But then I realized that I would have to write a separate compareKeyAsNumber, so instead I generalized it to:

Y.Sort.compareKeyAs = function(f, key, a,b)
{
    return f(a[key], b[key]);
};

Since sort requires a function that takes only (a,b), one must use Y.bind to fix the first two arguments:

sort(Y.bind(Y.Sort.compareAsKey, null, Y.Sort.compareAsString, key))

So far, we have only considered functions which operate on functions. One can also build functions that return functions. A simple example is a function to reverse the sort order:

Y.Sort.flip = function(f)
{
    return function(a,b)
    {
        return f(b,a);
    };
};

Hopefully, these examples of using functional programming to generalize code will inspire you to look for situations where you can do the same in your own code.

About the author: John Lindal (@jafl5272 on Twitter) is one of the lead engineers constructing the foundation on which Yahoo! APT is built. Previously, he worked on the Yahoo! Publisher Network.

Find conversations: bloglines | technorati

Share and extend: Bookmark with del.icio.us | digg it! | reddit!

Help Us Help Others!

One of the best things about YUI is our documentation. It’s been known in the community for years that documentation is a high priority for our developers. One of our other priorities is exceptional API documentation. We have always had high quality documentation but that comes with a price tag, time.

Did you know that you can help us with this?

All of our documentation is available in our Github repo and the tools we use to generate this documentation are available for you to use. Yes, that’s right, everything we use to make our documentation is available right now. In this article I will show you how you can fix API documentation and update an example to help other developers just like you.

This article is about modifying existing examples, landing pages and API documentation. Check out the screencast at the bottom if you’re more interested in creating new examples from scratch.

What do you need?

You need a working install of Node.js (0.6.x or higher is recommended), NPM (packaged with node), Selleck our custom documentation tool and YUIDoc our api documentation tool. These tools are freely available and very easy to install. Simply go here and choose your environment to install Node.js. Once it’s installed, you can install our two tools with this command:

npm -g install selleck yuidocjs

Once that completes, you’re all set!

** You will, of course, need git installed, have a Github account and have already forked the yui3 repo

Where things live

All of our examples are kept in our main source tree, this makes it easier to associate an example with the code it belongs to. In this example, I will be working with the DragDrop examples and API docs.

The main landing page and example files are located under yui3/src/dd/docs. All of the API docs are parsed from the raw source files (we’ll get into that in a bit).

Seeing a rendered example

The hardest part of any example is seeing how it looks once it’s ready. This is where Selleck comes in to play. Selleck has a “server mode” that you can fire up and see our examples “parsed and loaded”. Turning on Selleck’s“server mode” is very simple:

cd yui3/src;
selleck --server

This will print the following to the console:

[info] Selleck is now giving Ferrari rides at http://localhost:3000

Now visit http://127.0.0.1:3000 in your browser of choice and you should see the main Selleck page displaying a list of all the components that it found under the src directory.

If you don’t want Selleck to bind to port 3000, simply add a port to the command above (selleck --server 5000)

One of the advantages of using Selleck in server mode is that you do not need to restart the server (unless you add new json files for new examples) to see your changes. Just open a file, edit, save and reload! It’s that easy!

Fixing API Documentation

All of our API documentation is parsed directly from our source files yui3/src/dd/js with
YUIDoc. This makes reading them in the source files a little difficult (unless you can parse JSDoc tags and Markdown syntax in your head). Luckily YUIDoc also has a server mode to help you with this. Turning on YUIDoc’s server mode is just as easy as Selleck’s:

cd yui3/src;
yuidoc --server

This will print out some YUIDoc debugging info ending with:

info: (server): Starting server: http://127.0.0.1:3000

Now visit http://127.0.0.1:3000 in your browser of choice and you should see the main YUIDoc page listing all of the parsed API docs.

If you don’t want YUIDoc to bind to port 3000, simply add a port to the command above (yuidoc --server 5000)

YUIDoc’s server mode works a lot like Selleck’s in that you do not need to restart the server to see your changes. YUIDoc will reparse all of the source code on each page load and show you the updated API docs. Just open a file, edit, save and reload! It’s that easy!

Things to remember

Some things to remember when you are working on something you want to contribute:

• Always work in a branch git checkout -b mydocpatch
• Push to your branch git push origin mydocpatch
• Submit the Pull Request from your branch
• (optional) If you’re comfortable with git, we recommend working against yui/yui3′s “live-docs” branch

What to do after I update things?

As with anything else in YUI, once you update your files, simply commit them and issue us a Pull Request as usual. One of our developers will verify the changes and either merge them in or give you some feedback.

How often are things updated on the site?

Our current site deployment is very easy, we often deploy to yuilibrary.com at least once a week. Sometimes we even push daily! So, get your changes in and give back to the community!

More Information

YUI Engineer Luke Smith put together the following screen cast showing how to create a new example from scratch. Take a look at it and more videos over on our YouTube Channel.

Find conversations: bloglines | technorati

Share and extend: Bookmark with del.icio.us | digg it! | reddit!

YUI: Open Hours Thurs Apr 26th

Code Review: Photos Near Me

In Eric F’s most recent article he reports in on his exercise to add YUI to the server side of his project app, Photos Near Me. Short form: so far, so good.

It’s been a while since we’ve done a good old fashioned code review, and this seems like a perfect candidate for it. Server side YUI, App framework, desktop and mobile considerations, progressive enhancement. Lots of good takeaways here.

Recording

The recording is available in the YUILibrary YouTube channel.

Find conversations: bloglines | technorati

Share and extend: Bookmark with del.icio.us | digg it! | reddit!

Using Your App’s YUI Components on the Server

For my first sprint of 3.6.0 development I’m writing up (and showing by example) how to develop an app using YUI on the client and server, which works on both the desktop and mobile devices. Code sharing and reuse FTW! To start this process, I wanted to first build something using the development approaches that we want to promote. I’ve done that and I want to report in on my first experience running YUI on Node.js (spoiler: it was awesome!)

Photos Near Me, an app that shows you interesting photos near your current location, is a project I started almost a year ago to dogfood the YUI App Framework while Ryan Grove and I were writing it. I’ve been keeping the app up to date with the latest changes and additions to the App Framework, including the use of Y.App, the newest component in the App Framework. Photos Near Me has always been a client-side only app up until now!

My plan was to power the Photos Near Me server using Node.js, Express.js, and YUI, and I set two goals: share the app’s model objects and share its Handlebars templates between the client and server. Thanks to Dav Glass who has put in a ton of effort to make YUI run on Node.js in an intuitive, seamless way, I was able to easily accomplish these goals after several hours of refactoring the app. I was pleasantly surprised that my first try instantiating one of the model objects server-side and calling its load() method, which fetches data from YQL, just frickin’ worked!

Photos Near Me now renders the initial state for a request on the server then hands off control to the client-side Y.App instance. From there, in modern browsers which can use HTML5 History, all routing, data fetching, and page rendering will be done client-side; while older browsers will perform full page loads handled by the server. The time from request to seeing photos has been drastically reduced, it was especially noticeable on my iPhone when refreshing the page.

Being able to use the exact same code for the models and templates between the client and server makes following the best practices of progressive enhancement much easier to implement.

Check out Photos Near Me:
http://photosnear.me/

And its source:
https://github.com/ericf/photosnear.me

Watch for my comprehensive tutorial on building apps with YUI which will use Photos Near Me as an example to describe and show in detail how to use YUI on the server and client to build apps which run in desktop and mobile browsers while following the best practices of progressive enhancement.

Find conversations: bloglines | technorati

Share and extend: Bookmark with del.icio.us | digg it! | reddit!

Migrating from YUI 2 to YUI 3: A Case Study

When I sat down to build the YUI 3 version of Page Layout, I knew it would be a big job. Even though the YUI 2 version was on its third incarnation, the code was still a mess. The original design, dramatically simplified from the performance disaster of the second incarnation, called for only row-based layouts, but then somebody needed a column-based layout, and they needed it fast, so instead of refactoring the code cleanly, I duplicated it and rewrote only the layout engine. In addition, I knew that the only way to get people to migrate from the YUI 2 version to the YUI 3 version would be to maintain exactly the same API in the YUI 3 version, so nobody would have to rewrite their code. To make matters worse, the YUI 2 version was a global object that automatically instantiated itself when the script loaded, so application code could configure it and subscribe to events before domready. The worst implication of this was that the initial object assumed a row-based layout, and then, on domready, if a column-based layout was detected, the object silently replaced itself, copying the settings from the old object! (The row and column versions shared the same event objects, so the subscribers were not affected.)

With all this to consider, the first thing I did was to ignore it all because I wanted to get the YUI 3 version right. I was satisfied with the basic API, but the two copies and the silent switching on domready had to go. Inspired by the YUI 3 Plugin architecture, I decided that the ideal solution would be to have a single class which detected the layout type and used the corresponding layout engine. The YUI Loader would even allow me to load the layout engine on demand! It took a couple of weeks to shred the two YUI 2 classes and merge them into a single class with plugins, but the result was clean and (as long as you don’t look too closely at the code in the layout engine plugins) simple.

Now came the hard part: how to make this a drop-in replacement for the YUI 2 version. The applications that use it still have lots of YUI 2 code, and this cannot reference Y, so YAHOO.APEX.PageLayout had to be defined, it had to be created when the script loaded, and it had to expose all the required functions and event signatures. To muddle things further, YUI 3 event signatures are fundamentally different from YUI 2 event signatures.

There was also another serious complication: YUI 3 doesn’t really like global objects. Everything is normally confined to the Y sandbox created by YUI().use().

The first step was to break the sandbox by storing the instance of Y.PageLayout in YUI.SATG.prototype.layout_mgr. (By instantiating YUI.SATG, each sandbox starts with the same initial values and can then safely modify Y.SATG without affecting other sandboxes. Of course, overwriting Y.SATG.layout_mgr would be a bad idea, but there is other stuff in this object, too.)

Breaking the sandbox is not something to be done lightly, however. The most dramatic problem is that instanceof does not work on objects passed between sandboxes. This is disasterous for Y.PageLayout.elementResized(), since the argument, an instance of Y.Node, is likely to come from a sandbox other than the one where Y.PageLayout was instantiated. Thankfully, YUI 3.5.0 switched from instanceof Y.Node to testing for the _node member!

The next step was to define YAHOO.APEX.PageLayout (but only if YAHOO already existed, of course). This object turned out to be very thin, since it only had to act as a relay. It stores references to functions in Y.PageLayout, including a couple of renames, and instances of YAHOO.util.CustomEvent.

The final step was to subscribe to the events from Y.PageLayout, repackage the data, and fire the corresponding events in YAHOO.APEX.PageLayout. As an example:

page_layout.on('beforeResizeModule', function(e)
{
	YAHOO.APEX.PageLayout.onBeforeResizeModule.fire(e.bd, e.height, e.width);
});

I hope this overview of the challenges I faced will inspire you, or at least make the task seem less daunting, when you to migrate to YUI 3.

About the author: John Lindal (@jafl5272 on Twitter) is one of the lead engineers constructing the foundation on which Yahoo! APT is built. Previously, he worked on the Yahoo! Publisher Network.

Find conversations: bloglines | technorati

Share and extend: Bookmark with del.icio.us | digg it! | reddit!