YUI and Loader changes for 3.4.0

By YUI TeamJuly 1st, 2011

In 3.4.0 we started the process of shifting some of Loader’s logic around, to not only make it more performant, but to make it more robust and easier to use in other places (like on the server). We will be rolling out more changes in future revisions, but I wanted to take some time and explain what was changed, why it was changed and how it may impact developers. For the majority of use-cases, developers will notice nothing different, except that things are a little faster and their requirement downloads are a little smaller.

Seed File

The first thing I want to address is the YUI seed file. In previous versions of YUI, our seed file was very tiny and did not contain Loader or any of its meta-data. We’ve found that in the 90% use-case this was not as performant as we had hoped. The normal user includes the seed file then requests their modules, which in turn means that the seed needs to first fetch Loader, then calculate all of its dependencies, then fetch them all. We now feel that this extra http request is the wrong thing to do, so the new standard seed file contains Loader and its meta-data. Yes, this will make the initial request a little larger, but it will make the loading of modules that much faster since all of its meta-data requirements are now already on the page.

If you wish to use it the old way, you can just include the yui-base seed file instead. It contains everything that is needed to make YUI run in stand-alone mode plus it contains the ability to fetch Loader on demand. If you require even finer-grained dependencies, we have created a yui-core seed file that is exactly what the old yui-base seed was.

    /build/yui/yui-min.js //YUI Seed + Loader
    /build/yui-base/yui-base-min.js //Old YUI Seed with Loader fetch support
    /build/yui-core/yui-core-min.js //Old yui-base without Loader fetch support

It should be noted that these URLs are different than the previous URLs. Anyone that was using the yui/yui-base.js files need to repoint them to yui-core/yui-core.js. If you want the older way of loading the seed and fetching Loader, you would use the yui-base/yui-base.js seed file.

The other reasoning for this change is our roadmap for making YUI run in as many places as possible. The old seed file plus Loader in a single combo server request is all well and good if you have a combo server available in your application. But what about on the server? Or in an offline app on a mobile device? These places need to minimize file access while still getting the information they need.

Rollups

The next thing that we changed was removing rollups from the system and defaulting allowRollup to false in the Loader config. What does this mean to you? Well, hopefully nothing at all. Before I explain the impact of the change, let me explain the reasoning behind it. The primary reason is, again, performance, along with payload delivery. Take this example:

     Module A: requires event-a, event-b
     Module B: requires event-c, event-d

When you request both, the rollup logic prior to 3.4.0 used to determine that you should get the event rollup. Which actually meant that you were getting:

     event.a, event.b, event.c, event.d, event.e, event.f, event.g, event.h

You ended up with more on your page than you actually needed. By turning off the rollup support, YUI will now ask for only what you actually requested and nothing more. In most cases, you will not notice this. Module developers, may run into a situation where things that worked in the past may not work now. The reason for this is that they actually worked by accident before. Let me use a real world example: Dial.

In 3.3.0, Dial required this:

    requires: [
        'widget',
        'dd-drag',
        'substitute',
        'event-mouseenter', 
        'transition',
        'intl'
    ]

For the most part, Dial worked in 3.4.0, however keyboard support did not work. After doing some simple investigating, it turned out that the rollup support was actually requesting the entire Event rollup (which includes event-move and event-key). Without the rollup logic pulling in all of event, 3.4.0 Dial no longer had all of its requirements. Making Dial’s requirements more specific and defining all of its actual dependencies properly makes it work as expected.

    requires: [
        'widget',
        'dd-drag',
        'substitute',
        'event-mouseenter',
        'event-move',
        'event-key',
        'transition',
        'intl'
    ]

For module developers, it is a best practice to make sure that your module requires exactly what it needs to function. Do not assume that an upstream module requirement is there. It’s always better to make sure that you ask for what you need.

This also means that module requirements are more well defined. For example, datatype-date has Intl support built in. In previous versions you would access the Intl like this:

    Y.Intl.getAvailableLangs('datatype-date');

But since this module doesn’t actually have a language (the datatype-date-format module does), this will fail. It needs to be more specific and actually ask for languages for the correct module:

    Y.Intl.getAvailableLangs('datatype-date-format');

Build File Explosion and Submodule Removal

After making this change, the next change we made was exploding the build directory and removing submodules from the core system. Submodule logic was not removed, only our meta-data structure was changed. This will provide backward compatibility for current installations.

Submodules in the core system caused a couple of issues that we needed to resolve. The first reason was performance. Each time Loader needed to calculate dependencies, it needed to walk the submodule/plugin structure of each module. Doing this thousands of times was hurting our performance during the Loader calculate routine. By removing support for submodules in the core system we saved tens of thousands of function calls and iterations.

Loader was changed so that if a use property in a module’s meta-data defined more modules, it will use those modules instead of trying to load the original module. So, if you requested “dd” Loader would inspect “dd“‘s meta-data and see a use property that looks something like this:

    "dd-ddm-base,dd-ddm,dd-ddm-drop,dd-drag,dd-proxy,dd-constrain,dd-drop,dd-scroll,dd-drop-plugin"

In the core YUI seed file, we are also including what we are calling virtual rollups or aliases. These module definitions are exactly the same as the meta-data in Loader. This way you can include all the files exported from our Dependency Configurator and still use these rollups without having Loader present on the page. In future releases, we will be refining this approach even more.

After making this change, we then preceeded to explode our build files. In previous releases, the submodules determined the modules file path. For example:

    "dd": {
        "submodules": {
            "dd-drag": 
            // Module data
        }
    }

In 3.3.0 when you built “dd”, the file structure looked something like this:

    /build/dd/dd-drag.js
    /build/dd/dd-ddm.js
    /build/dd/dd-drop.js

With the build system exploded in 3.4.0, “dd”‘s build files now look like this:

    /build/dd-drag/dd-drag.js
    /build/dd-ddm/dd-ddm.js
    /build/dd-drop/dd-drop.js

This allowed us to remove the “path” property from all of our module meta-data as well, saving file size and reducing the logic required to assemble the modules url paths.

If you are including a pre-configured combo URL, you must recalculate your URL when you upgrade.

The downside to this change is that if you are including a combo URL of modules to “prep” your page you can not just change the version number and upgrade. You will need to revisit the Dependency Configurator and generate a new URL with new module structure.

The Future

I will be continuing to refine, refactor and maximize every aspect of our Loader and Seed strategy. These first steps were needed to aid in future changes that need to be made for not only our client-side strategy but also our server, command-line and mobile device strategies as well.

10 Comments

  1. I don’t quite get this passage:

    “Loader was changed so that if a use property in a module’s meta-data defined more modules, it will use those modules instead of trying to load the original module. So, if you requested “dd” Loader would inspect “dd“‘s meta-data and see a use property that looks something like this:

    “dd-ddm-base,dd-ddm,dd-ddm-drop,dd-drag,dd-proxy,dd-constrain,dd-drop,dd-scroll,dd-drop-plugin”

    Can you explain this telling by contrast what happens in the current version?

    Thx
    /paolo

  2. @Paolo

    In versions prior to 3.4.0 PR2, we created static rollup files. So in the case of DD, there was a file called:

    /build/dd/dd.js

    And that contained all of the DD related modules.

  3. I’m glad you guys are doing this. I always avoided roll ups like the plague. However, since some of the base classes seem to be removed now, e.g. node-base becomes node-core, it would be a good thing to get a list of all the classes that have been deprecated in this way.

  4. @Marc

    When 3.4.0 is released, there should be plenty of documentation around the changed modules.

  5. This comment is not to this post specifically, but I still find the philosophy of YUI against my belief in good design, “simple things should be simple, complex things should be possible”.

    For I’ve found myself again and again reading through documentation of the most fundamental modules like node & event, after months with YUI3.

    The new App/MVC modules seem perfect for my app at first, then I started writing lots custom logic to get around the default behavior due to huge memory overhead. When I throw away Module & View completely, I ended up with a much shorter and compact logic, only faster!

    Another case is context menu, which I didn’t find in YUI3. Some YUI experts suggest creating a new class based on Overlay then add various custom properties and rendering methods … I just need ONE context menu and eventually I got it done with less then 10 lines of raw JS + CSS.

    I never doubt YUI3 can support complex situations, but what about very SIMPLE cases, like a hello world app? With C, you just call printf(“hello world!”) within main(). For Java, you need to first define a HelloWorld class, then a public static void method called main(), and finally, call System.out.println(“hello world!”) … OMG, anyone realizes how much prerequisite knowledge it takes to write something as simple as this? That’s exactly how I feel about YUI!

    Whatsoever, I don’t think YUI3 will ever become simpler, but I still hope you guys can make it as simple as possible.

  6. @Wayne: You don’t need YUI to write a Hello World app. It sounds like you also didn’t need YUI to build the app you tried to build using the Model and View components.

    That’s okay!

    YUI isn’t going to be the perfect tool for every task. If you can write ten lines of JavaScript and accomplish what you need to do, then by all means, do that instead of using a widget that does far more than you need.

    This doesn’t mean that the widget is useless, or unnecessarily complex. It means it wasn’t the right tool for the job you wanted done.

    We absolutely want to make YUI as simple as is practical, but there’s a balance to be achieved between simplicity and usefulness. The simplest code is no code at all, but if we went that far, we also wouldn’t provide any value. So we aim for a balance: we try to make complex things simpler, but without oversimplifying them to the point where our abstractions aren’t useful.

    I agree with your philosophy that simple things should be simple and complex things should be possible. And I believe that’s exactly what YUI does: it makes the complex things possible without making the simple things any less simple. The beauty of YUI’s modular architecture is that if you find it simpler to roll your own code instead of using a particular widget, you’re not forced to load that widget code. And the Loader changes coming in 3.4.0 make this even simpler.

  7. [...] Loader has had a significant update for 3.4.0. If you are doing manual load specifications via use("*") or make use of submodule configurations, we’d greatly appreciate you trying your code with the new loader to be sure we are correctly handling all use cases. For more detailed information on the Loader changes in this release, refer to the blog post describing 3.4.0 Loader changes. [...]

  8. @Ryan,

    Just saw your comment. I was maintaining an old project written on YUI 3.1 with 1000s of lines of code, and I just wanted to add ONE context-menu. But I failed to find any feasible module / method to get it done quick and easy. Foreseeing this tiny feature might require 100s of lines of YUI oriented code, I used raw JS instead.

    BTW: there were few glitches when switching to 3.2, but the entire system stopped working when switching to 3.3. So I don’t think it necessary to try 3.4 for this project.

  9. [...] the number of iterations executed by Loader in the calculation of dependencies. You can refer to the blog post about Loader changes in 3.4.0 for more [...]

  10. All of these changes to the loader seem very promising and the removal of sub-modules calculations is great. That was probably the reason we were seeing tons of function calls being made in IE when doing dependency calculations and IE would sorta choke for a few seconds.

    With the changes to the loader and the meta data — how is the yui server side loader affected. Is that being updated to a take into account of this changes. Cause thats what we ended up using.