Treeble: Using Nested YUI 2 DataSources for Row Expansion

April 14, 2010 at 8:00 am by John Lindal | In Development | 32 Comments

John Lindal (@jafl5272 on twitter) is one of the lead engineers constructing the foundation on which Yahoo! APT is built. Previously, he worked on the Yahoo! Publisher Network.

Daniel Barreiro’s recent post about nested tables reminded me that it was about time I finished my “treeble” widget. “Treeble” comes from merging “tree” and “table.” The original motivation was to enable drilling into the details behind each row in a table, e.g., start with a table which displays sales figures for each continent and then drill into each country, region, and city. Of course, once I started building it, the natural design was to support any hierarchy of data. As an example, it can display a tree like this:

(Click the screenshot to play with this example.)

Overview

The treeble widget consists of two parts. The first is a small extension to YUI DataTable for opening/closing nodes in the tree. (This is done by augmenting DataTable in order to allow it to work transparently with existing extensions to DataTable, most notably ScrollingDataTable.) The second part of treeble is a significant extension to YUI DataSource, called TreebleDataSource, which merges the results from simple, flat data sources. The tree is built dynamically, so you don’t have to load all your data at once.

The tricky part is paginating the tree: knowing the subset of nodes which have been opened, request only the visible items. TreebleDataSource provides two options:

Paginate top-level nodes only, so all their children can be visible on the same page (the default)
Paginate so a fixed number of nodes are visible

#1 is ideal when the tree is shallow and there are only a few children per node. #2 is necessary when there may be a very large number of children per node. The configuration option that controls this behavior is paginateChildren. In order for pagination to work, the total number of items available from each simple data source must not change.

Since all the items from all the simple data sources are displayed in a single table with a single set of columns, the schema given to DataTable must be the union of all the flat data source schemas. For example, in the above screenshot, DataTable must have a column for Quantity even though it is only defined for the leaf nodes.

Obviously, rows in the table no longer map directly to records in the simple data sources. Instead, each record in the DataTable has three special members:

_yui_node_ds: The DataSource from which the record was retrieved. If you allow inline editing, this tells you which DataSource to use when saving the new value. (You can edit the Quantity column in the live example.)
_yui_node_depth: The depth of the node in the tree. Top-level nodes have depth zero. This is useful when indenting child nodes, as in the above screenshot.
_yui_node_path: Array of node indices leading to the record. For example, [2,5,1] translates to second child of the sixth child of the third top-level node. DataTable.rowIsOpen() and DataTable.toggleRow() require this array to identify the node.

The YUIDoc for TreebleDataSource and the extensions to DataTable is here.

Configuring DataTable

To work with TreebleDataSource, your YUI DataTable must be configured correctly:

var myDataSource = new YAHOO.util.TreebleDataSource(...);
var myDataTable = new YAHOO.widget.DataTable(container, columns, myDataSource,
{
  dynamicData:true,
  generateRequest: YAHOO.widget.DataTable.generateTreebleDataSourceRequest,
  displayAllRecords: the opposite of your TreebleDataSource's paginateChildren value,
  other configuration, e.g., paginator
});

The special version of generateRequest is required because TreebleDataSource needs to receive a known data format so it can correctly generate the requests to the individual simple data sources.

Constructing TreebleDataSource

When you construct TreebleDataSource, you must pass it an instance of YUI DataSource as oLiveData. One column in the data schema for this data source must have its parser set to either "datasource" or a custom parsing function which does the same job: construct a new YUI DataSource from the value in the column. The default data source parser ignores falsey values and constructs a data source from any truthy value, e.g., an array of objects, an XHR URL, or an object which defines dataType and liveData. The configuration (including responseSchema and treebleConfig) for the new data source is copied from the parent data source. Writing a custom datasource parser is discussed later in this post.

TreebleDataSource itself has only one configuration parameter, paginateChildren, which controls the pagination behavior, as discussed earlier. The majority of the treeble-related configuration is set on the simple data sources via the treebleConfig object. This allows the configuration to be different for each simple data source, as discussed below in the section on mixed data sources.

Using local DataSources

The live example demonstrates how to work with local DataSources. The only required values in treebleConfig are generateRequest and totalRecordsReturnExpr:

new LocalDataSource(array,
{
  ...,
  treebleConfig:
  {
    totalRecordsReturnExpr: '.meta.totalRecords',
    generateRequest: function(state, path)
    {
      return state;
    }
  }
});

Since LocalDataSource ignores all request parameters, generateRequest could actually return null, but it echoes the state object instead to support extensions which can sort the data.

Setting totalRecordsReturnExpr signals TreebleDataSource that the simple data source will return all its records, not just the requested slice. The actual value of totalRecordsReturnExpr must be an OGNL expression specifying where in the oParsedResponse object returned by TreebleDataSource to store the total number of visible nodes, based on which nodes are currently open. In the example, this is oParsedResponse.meta.totalRecords. Since DataTable has (and must have!) dynamicData set to true, the live example also overrides DataTable.handleDataReturnPayload to set oPayload.totalRecords=oParsedResponse.meta.totalRecords. This gives the paginator the information it needs to compute the total number of pages.

Using XHR DataSources

An example configuration when using XHR DataSources would be to construct the top-level data source as:

new XHRDataSource(url,
{
  ...,
  treebleConfig:
  {
    startIndexExpr: '.meta.startIndex',
    totalRecordsExpr: '.meta.totalRecords',
    generateRequest: function(state, path)
    {
      return 'path='+path.join(',')+
                 '&startIndex='+state.startIndex+
                 '&results='+state.results+
                 '&sort='+state.sort+
                 '&dir='+state.dir;
    }
  }
});

In this example, generateRequest returns query args which the server will interpret in order to return the appropriately sorted slice of the children of the node specified by path.

The value of startIndexExpr is an OGNL expression specifying where in the oParsedResponse object returned by the simple data source the index of the first returned node is stored. In the example, this is oParsedResponse.meta.startIndex.

The value of totalRecordsExpr is an OGNL expression specifying where in oParsedResponse the total number of nodes is stored. In the example, this is oParsedResponse.meta.totalRecords. totalRecordsExpr also specifies where in the oParsedResponse object returned by TreebleDataSource to store the total number of visible nodes, based on which nodes are currently open.

Since DataTable has (and must have!) dynamicData set to true, you would also have to override DataTable.handleDataReturnPayload to set oPayload.totalRecords=oParsedResponse.meta.totalRecords. This gives the paginator the information it needs to compute the total number of pages.

Using Mixed DataSources

TreebleDataSource does not require that all the simple data sources be the same type. For example, if you have a large number of top-level nodes, but only a small tree of children for each node, then it makes sense to return the entire tree when a top-level node is opened. The default data source parser actually handles this automagically if you specify startIndexExpr, totalRecordsExpr, and totalRecordsReturnExpr for the top-level data source!

If you have only a few top-level nodes, but each tree of children is huge, then your top-level data source could use local data, and you could build a custom data source parser which instantiates XHR data sources for the children, setting startIndexExpr, totalRecordsExpr, and generateRequest appropriately.

You are only limited by your ability to comprehend the complexity of the system!

Note that, when using a custom data source parser, you must define childNodesKey in treebleConfig for each simple data source so TreebleDataSource knows the name of the data source column. (When you use the default parser, this is detected automatically.)

Share and extend: Bookmark with del.icio.us | digg it! | reddit!

32 Comments

I’ve found a bug in your clickable treeble demo (http://jafl.github.com/yui2/treeble/) where the CS Lewis row becomes duplicated – to reproduce:

1) Click Fantasy
2) Click CS Lewis
3) Click Philip Pullman

There is now a second CS Lewis row above Philip Pullman. You can click it to collapse the CS Lewis subset, or you can click/collapse Philip Pullman to make the second CS Lewis row disappear.

Perhaps an error in your source data?

Comment by Marcus Tucker — April 14, 2010 #
As an addition to Marcus, it also happens on page 2, again with the first item.

Comment by Jeff Craig — April 14, 2010 #
I don’t know how I missed that. I have updated the code to fix it.

Comment by John Lindal — April 14, 2010 #
I hope YUI3 will include a tree table. This is the #1 reason i’m torn between YUI and Dojo.

Comment by mikerobi — April 15, 2010 #
Nice work John!

Comment by Paul — April 15, 2010 #
@mikerobi:

You can use Treeble with YUI 3 immediately via the 2-in-3 functionality built by Dav Glass in YUI 3.1.

Once YUI 3 has a native DataTable, I will port Treeble to YUI 3.

Comment by John Lindal — April 15, 2010 #
@John

I meant as part of the YUI built in widgets.

Comment by mikerobi — April 15, 2010 #
@mikerobi: I have to leave that decision to the core YUI team, but I’m hoping it will make it into core!

Comment by John Lindal — April 15, 2010 #
@mikerobi — Speaking on behalf of the YUI core team, we hear you. It will either be part of the shipping widget or someone like John will build a great extension for the YUI 3 Gallery, which would be equally useful. You’ll have options. -Eric

Comment by Eric Miraglia — April 15, 2010 #
Wow, just what I was looking for — thanks!

But is there a way to add sorting to the top-level nodes in the table? For example, is it possible to sort by title before drilling down to child nodes?

Comment by J. Atwood — May 5, 2010 #
Sorting must be done by the DataSource. This is easy when using XHR, since the server simply sorts before returning the data. Since each set of children is loaded separately, the tree structure will be maintained. When using a local DataSource, you will need to build some custom JavaScript, since vanilla YUI LocalDataSource doesn’t sort.

Comment by John Lindal — May 5, 2010 #
@mikerobi — TreebleDataSource is now available in the YUI 3 Gallery: http://yuilibrary.com/gallery/show/treeble

Comment by John Lindal — June 7, 2010 #
Nice work John. One query, is it possible to highlight the selected/expanded node?

Comment by Amitabh — June 9, 2010 #
@amitabh: Yes, your click handler (installed in the cell formatter) can save the row that was toggled, and then the cell formatter can apply a css class to hilight the row when the table is re-rendered after opening/closing the row.

Comment by John Lindal — June 9, 2010 #
Hi,

It is a very good work. Thank you.

But there is still the bug that Marcus mentioned but this time from the second page when there are more than 2 items. (except for the last item)

Comment by Thanh — June 16, 2010 #
@thanh: I’m not able to reproduce your claim. Are you testing for YUI 2 or YUI 3? Please provide the exact sequence of steps that you did to get the incorrect behavior.

Comment by John Lindal — June 17, 2010 #
Hi,
Actually, I found the same bug either with YUI 2 and YUI 3.
So here are the steps to reproduce :

1 – use a sample data having more than 8 top-level nodes containing children
2 – select 5 rows per page
3 – go on page 2
4 – click on the first row of page 2

-> We will see that all row (top-level nodes) above are duplicated

Here is the example that I used :
http://www.chillythanh.com/treebleTesting.html

Comment by Thanh — June 17, 2010 #
Thanks! That’s what I needed.

This is what I get for whipping out a second algorithm just before I release the widget. The bug only manifested with paginateChildren:false. I’ve fixed the bug and enhanced my example to show that it works:

1 – select 2 rows per page
2 – go to page 2
3 – click on the first row

With the old code, it breaks, but the updated code works. I have fixed both the YUI 2 and YUI 3 examples. I have also scheduled a new cdn push for the YUI 3 Gallery module.

Comment by John Lindal — June 17, 2010 #
Great ! It works now !!
Thank you so much.

Your widget really comes at the right moment ’cause I’ve been looking for this kind of solution working with YUI2, but I did’nt find anything so far except some functions with jquery treetable.

I’m so grateful, thanks!! :-)

Comment by Thanh — June 17, 2010 #
I’m intrigued by this statement.

“Since LocalDataSource ignores all request parameters, generateRequest could actually return null, but it echoes the state object instead to support extensions which can sort the data.”

I’m trying to implement client-side sorting and am wondering what exactly I should be extending. Has anyone put together a working example?

Thanks!

Comment by J. Atwood — July 1, 2010 #
Create a LocalDataSource and override makeConnection() to sort the data array before returning it. In order to do this for TreebleDataSource, you need to provide a custom parsing function (instead of specifying parser:’datasource’) and configure childNodesKey on the root data source. This will ensure that every child data source will also be sorted.

Comment by John Lindal — July 1, 2010 #
Thanks for your help, John!

I was able to override makeConnection() and now the top level sorts perfectly.

The one remaining thing I’m having trouble with is that now the toggle to expand a top level node only works when the table is in the original sort order.

Example: http://web.me.com/jeremiah.atwood/sort_example/sort_example.html

Click the Title header once to sort ascending, then expand Science Fiction. Isaac Newton shows up under Science Fiction…

- Jeremiah

Comment by J. Atwood — July 2, 2010 #
The solution you found for sorting in makeConnection() is very nice! I’ve generalized it and added it to the example.

I’ve updated TreebleDataSource for both YUI 2 and YUI 3 with a simple, ugly fix to prevent incorrect results from being displayed: when you change the sort, everything closes. Unfortunately, this is the best one can do without more information.

To provide a nicer user experience, I’m going to introduce a new configuration: uniqueIdKey. If specified, this will be used to re-sync the internal state after sorting instead of throwing it all away. Stay tuned!

Comment by John Lindal — July 2, 2010 #
Hi John,
I m working with a project which using YUI widget a lot. Recently our project comes with a requirement of treegrid like your treeble widget. Could you please get me a example using XHR DataSources.
thanks in advance!

Comment by Jerry Cai — July 15, 2010 #
There have been some questions about how to use an XHR datasource. Here is an example:

var paginateChildren = true;

function xhrGenerateRequest(state, path)
{
return “startIndex=” + state.startIndex +
(state.results !== null ? “&results=” + state.results : “”) +
(state.sort ? “&sort=” + state.sort : “”) +
(state.dir ? “&dir=” + state.dir : “”) +
(path && path.length > 0 ? “&kiddies=” + path.join(“-”) : “”);
};

var root_ds = new YAHOO.util.XHRDataSource(‘treeble_data.php?’,
{
responseSchema:
{
resultsList: ‘records’,
fields: [ "id","quantity","amount", {key: 'kiddies', parser: 'datasource'} ],
metaFields: { startIndex: ‘first_index’, totalRecords: ‘total_records’ }
},
treebleConfig:
{
generateRequest: xhrGenerateRequest,
startIndexExpr: ‘.meta.startIndex’,
totalRecordsExpr: ‘.meta.totalRecords’
}
};

var ds = new YAHOO.util.TreebleDataSource(root_ds, { paginateChildren: paginateChildren });

var config =
{
initialRequest: {startIndex:0,results:5},
dynamicData: true,
generateRequest: YAHOO.widget.DataTable.generateTreebleDataSourceRequest,
displayAllRecords: !paginateChildren,
handleDataReturnPayload: function(oRequest, oResponse, oPayload)
{
oPayload.totalRecords = oResponse.meta.totalRecords;
return oPayload;
}
};

var table = new YAHOO.widget.DataTable(id, cols, ds, config);

Comment by John Lindal — July 19, 2010 #
I have updated the YUI 2 code to support restoring state after sorting. The new configuration is “uniqueIdKey.” See the docs for all the details: http://jafl.github.com/yui2/treeble/yuidoc/YAHOO.util.TreebleDataSource.html

The YUI 3 version is waiting for YUI 3.2.0 to be released, because that will enable significant simplification of the configuration required to construct TreebleDataSource.

Comment by John Lindal — August 11, 2010 #
Awesome! Treeble is hands-down the best option I’ve found for TreeGrids.

Thanks,
J

Comment by J. Atwood — August 12, 2010 #
Just wondering if the license is the same as YUI.

Great work,

Joao

Comment by Joao Leal — August 30, 2010 #
Yes, it’s BSD. I updated the files to make this explicit.

Comment by John Lindal — August 30, 2010 #
The YUI 3.2.0 version of Treeble is now available:

http://jafl.github.com/yui3-gallery/treeble/

The code required to configure the treeble is simpler (thanks to changes in YUI 3.2.0) and Treeble itself now runs much faster!

Comment by John Lindal — September 8, 2010 #
Just found an important difference between treeble and a normal datatable when using a XHR datasource.

In a normal datatable using a JSON datasource the results for a given page do not have to have their indexes matching their position in the datasource, they only have to be in right order for that page.

Example:
A datasource requests results for page 2 with the results position from 20 to 39. The response could be something like:
{results: [
{"row": 20, "zoot":"a"},
{"row": 21, "zoot":"b"},
...
{"row": 39, "zoot":"z"},
]
}

The datatable would render the results for that page using this response.

Treeble datasource expects each result to have their indexes matching the datasource. Therefore, the elements from response.results[20] to response.results[39] have to be set. The response in the above example only has elements from 0 to 19.

Comment by Joao Leal — October 21, 2010 #
This issue can be solved by setting ‘startIndexExpr’ in treebleConfig and then by either setting this value in response or by making use of responseParseEvent in the datasource.

Comment by Joao Leal — October 21, 2010 #

Sorry, the comment form is closed at this time.

Yahoo! User Interface Blog (YUIBlog)