Node.js Essentials: Master the “require()” Function by Constructing a Homemade Module Loader

What is a module?

A module is a block of code in a file. Modules are the bricks for structuring applications that are too big to efficiently maintain in a single file. They are the main mechanism to enforce information hiding by keeping private all the functions and variables that are not explicitly exported.

The structure of these modules in Node.js is based on the CommonJS module specification. This specification addresses how modules should be written in order to be interoperable among a class of module systems that can be both client and server side. CommonJS modules can be recognized by the use of the require() function, the exports and the module objects. These three dependencies are automatically injected into every module.

A module in Node.js looks like:

const dependency = require('./anotherDependency')

// A private function
function log() {
    console.log(`Hello ${dependency.username}`)
}

// run is a function exported for public use
module.exports.run = () => log()

A homemade module loader

The require() function is used to import modules, JSON, and local files. To explain how the require() function works, let’s build a similar system from scratch. Let’s start by creating a function that loads the content of a module, wraps it in a private scope and evaluates it:

/**
 * Creates a private scope and executes a module
 * @param {string} filepath 
 * @param {{ exports: {}}} module 
 * @param {function} require 
 */
function loadModule(filepath, module, require) {
    /**
     * The 3 CommonJS dependencies are passed into
     * the module by the wrapping function.
     * Notice how the exports parameter is assigned a
     * reference to the module.exports object
     */
    const wrappedSourceCode = '(function wrapper(module, exports, require) {'
        + fs.readFileSync(filepath, 'utf-8') +
    '}(module, module.exports, require))'

    eval(wrappedSourceCode)
}

The source code of the module is wrapped in an immediately invoked function expression and executed. The module exports its public API’s on the module and exports objects.

Let’s see what the objects, module and exports contain by implementing our require() function:

/**
 * @param {string} moduleName 
 * @returns {any}
 */
function require(moduleName) {
    const id = require.resolve(moduleName) // [1]
    
    if (require.cache[id]) { // [2]
        return require.cache[id].exports
    }

    const module = { // [3]
        exports: {},
        id
    }

    // Update cache
    require.cache[id] = module // [4]

    loadModule(id, module, require) // [5]

    // Return exported variables
    return module.exports // [6]
}

require.cache = {}
require.resolve = () => {
    // Resolve a full module id from moduleName
}

1. A module name is accepted, and we resolve the full path of the module and call it id. This is done by the require.resolve algorithm, which we’ll examine later.
2. If the module was already loaded in the past, it’ll be available in the cache. In this case, we just return it immediately.
3. If the module was not yet loaded, we create an environment object, module for it. The object holds the id of the module and the exports object which will be used by the code of the module to export public API’s.
4. The module object is cached
5. Load the module by passing in the id, a reference to the module object and the require function.
6. Return the public API of the module to the caller

As we see, by building our own module loader, we’ve unravelled the internal behaviour of the real require() function.

module.exports vs exports

From the loadModule function above, it is clear the exports variable is just a reference to the initial value of module.exports. This value is a simple object literal created before the module is loaded.

We can attach properties to the object referenced by the exports variable,

exports.hello = () => console.log('Hello') 

But reassigning the exports variable does not have any effect, because it doesn’t change the contents of module.exports, it will only reassign the variable itself. Therefore,

exports = function hello() { console.log('Hello') }

is wrong. It does not export the hello function.

If we want to export something other than an object literal, reassign module.exports,

module.exports = function hello() { console.log('Hello') }

The resolving algorithm

As we saw in the require function, require.resolve takes in a module name as input and returns the full path of the module. This path is then used to load the contents of the file. The algorithm has three major branches,

1. File modules: If the module name starts with a “/”, it is considered as the absolute path to the module, and it is returned as is. If the name starts with a “./”, it is considered a relative path, which is calculated starting from the requiring module.
2. Core modules: If the module name is not prefixed with a “/” or a “./”, the algorithm first searches for the module within the core Node.js modules.
3. Package modules: If no core module with the given module name is found, the search continues by looking for a matching module into the first node_modules directory (where npm installs dependencies) that is found, navigating up in the directory structure starting from the requiring module. This search continues in each node_modules directory up in the directory tree until it reaches the root of the file system.

For file and package modules, both individual files and directories can match module name. The algorithm will try to match the following,

1. <moduleName>.js
2. <moduleName>/index.js
3. The directory or file specified in the main property of <moduleName>/package.json

Consider this directory structure,

Here, myApp, depB and depC all depend on depA. However, they all have their private versions of the dependency!

Following the rules of the resolving algorithm, require('depA') will load a different file depending on the module that requires it,

1. Calling require('depA') from /myApp/foo.js will load /myApp/node_modules/depA/index.js
2. Calling require('depA') from /myApp/node_modules/depB/bar.js will load /myApp/node_modules/depB/node_modules/depA/index.js
3. Calling require('depA') from /myApp/node_modules/depC/foobar.js will load /myApp/node_modules/depC/node_modules/depA/index.js

🤔 Which file would be loaded if require('depA') is called from /myApp/node_modules/depC/foobar.js and/myApp/node_modules/depC/node_modules is empty? Leave your answer below.

The module cache

Each module is loaded and executed only the first time it is required. Subsequent calls to require()will return the cached version. The module cache is exposed in the require.cache variable, so it is possible to directly access it if needed.

Conclusion

In summary, delving into the require() function by creating your own module loader is more than just an academic exercise. This hands-on approach demystifies the internal mechanics of Node.js, enabling you to appreciate the elegance and efficiency of its module system. These insights provide a solid foundation for more advanced Node.js development, such as creating complex applications, optimizing performance, and even contributing to the Node.js core.

Don’t let the noise of others’ opinions drown out your own inner voice.

Steve Jobs