icon-search
6087358_startup-photos1

Building multi module projects with Gulp

Przemek Dziuba 24.06.2016

In this article, we’ll discuss the configuration of the Gulp building system. The article will be based on a real project we developed on Sparkbit. The main reason for writing this article is our frustration. Not a joke, honestly.

As our projects get bigger, our building process gets more and more complicated. And when we tried to find some help on the internet, all we found were tutorials and “Hello world” examples. There are no standard solutions for building big applications in JS. Each project I’ve seen has some custom scripts and tools.

We developed a rapidly growing front-end project for a couple of months, and here is what we came up with. It may or may not work for you. If you like splitting your code into modules and use Angular 2 and Typescript then there’s a big chance it will. Anyway, I hope you will find these general ideas useful and they’ll inspire you to look further into npm and Gulp dark secrets.

Introduction to Gulp

Gulp is a tool commonly used to automate the building process of an application written in node js. As the project gets bigger and standard npm scripts become insufficient, Gulp comes to the rescue. Its main advantages are its large number of plugins, concise format of config files, and its high performance brought by stream based file processing. It should be no surprise that gulp has become the most popular automation tool in the nodejs world. I think I’ve provided enough free advertisement to them. You can read more about basics of gulp on its official website here.

Typical configuration of build process

A typical gulp project includes a single gulpfile.js with a list of tasks. These “tasks” are usually things like compilation/transpilation of javascript, validation of code by some kind of linter, building css from sass, minification of scripts, and so on. If you’re developing a small sized project, it’s good to keep the whole configuration in a single file. As the project grows bigger, several shortcomings start developing. With big projects, the codebase gets divided into multiple modules and the list of tasks get longer. At Sparkbit, we’re currently developing a project consisting of a number of gulp modules. There’s a module with shared UI components, a module with some generic utilities, and some closed source modules. Each of these modules contains about 15-20 gulp tasks. Tasks across modules are pretty similar (well there’s a finite way of building typescript and angular projects). We started by simply copy-pasting the gulp configurations to each new module and adjusting file paths and output directories, and removing/adding snippets of javascript specific to each module. This is tedious work and very error-prone. When a bug in the configuration is found, it generally has to be fixed in 5 places. Any change in bundling or change of typescript plugin means changing the code in multiple places. It got complicated and we had to come with a better idea. We’ve defined a couple of goals we want to achieve:

  • Reduce code duplication
  • Allow simple enabling/disabling of any gulp task
  • Allow any module to be built independently or as a submodule of a bigger parent
  • Keep the building process as simple as possible, ideally with one command

Preparation steps

We started by looking at patterns and unifying the structure of our modules. Most of our projects are applications written in typescript, using AngularJS 2. Each project contains some html templates, sass styles, and static assets. So we’ve decided to use the constant structure of directories in all projects as given below:

└── project
├── built
│   └── result of the built process goes here
├── node_modules
├── src
│   ├── images
│   ├── main
│   │   ├── components
│   │   ├── i18n
│   │   ├── models
│   │   └── services
│   ├── styles
│   │   └── sass files
│   ├── test
│   │   └── unit tests
│   └── typings
│      └── tsd typings
└── test
└── karma/jasmine configuration

When all projects have the same structure, writing reusable gulp tasks is much easier as we’re using the convention over configuration approach. This way, you know all the typescript files will be there in src/main, sass files in styles directory, tsd files in typings directory and so on.

Splitting gulpfile into smaller pieces

We have split our monolithic gulpfile into a collection of small files. In this approach, each gulp task resides in separated commonjs module which exposes the interface as below:

{
 	//init method taking referance to gulp instance and object with build options
 	_init_method(gulp, options),
 	isWatchTask: boolean,
 	//does watch task needs to run dependencies? (see below for explanation)
 	watchTriggersDependencies: boolean,
 	//array of strings being file globs
 	srcFiles: function(): string[],
 	//array of strings with names of tasks that needs to be run earlier
 	dependencies: string[],
 	//function with actual task body, must return promise or run callback
 	task: function (callback): promise
 }

Tasks were made as generic as possible, to make them reusable in many modules. Any project specific constants were removed or replaced by parameters with sensible default value.

You can see an example of task – tslint.js below:

module.exports = function(gulp, options) {
	var TS = 'src/main/**/*.ts';
	return {
		isWatchTask: false,
		watchTriggersDependencies: false,
		dependencies: ['clear'],
		srcFiles: function() {
			return TS;
		},
		task: function(done) {

			var tslint = require('gulp-tslint');

			return gulp.src(TS, {
				cwd: options.cwd
			})
			.pipe(tslint())
			.pipe(tslint.report('verbose'));
		}
	}
};

As you can see, we’ve used cwd options of gulp telling where the working directory is. With this option, we can call the task from different directory – and use it in different modules.

Directory scan mechanism

Next, we created a piece of code which scans the gulp-tasks directory for js modules that implement the above interface and register gulp tasks. Since gulp has one “global namespace”, task names were prefixed with module names to avoid names clashes.

module.exports.createGulpModule = function(gulp, options) {

	var fs = require("fs");
	var MODULE_NAME = options.moduleName;
	if (MODULE_NAME === undefined) {
		throw "Configuration error - module name is not specified!";
	}
	if (options.dest === undefined) {
		options.dest = options.cwd + '/built';
	}

	//search in current module and in location given by client
	var directoriesToSearch = [__dirname, options.cwd];
	for (var dirIndex = 0; dirIndex < directoriesToSearch.length; dirIndex++) {
		var currentDir = directoriesToSearch[dirIndex] + '/gulp-tasks';
		if (!fs.existsSync(currentDir)) {
			continue;
		}
		console.log('Reading task definitions from ' + currentDir);
		var taskFiles = fs.readdirSync(currentDir);

		var tasksToRun = [];

		//put all tasks in 'namespace'
		for (var i = 0; i < taskFiles.length; i++) {
			var taskName = taskFiles[i].replace(/\.js$/, '');

			var taskModule = require(currentDir + '/' + taskName)(gulp, options);
			var taskDependenciesLocalNames = taskModule.dependencies;
			var taskDependenciesQualifiedNames = [];
			if (taskDependenciesLocalNames === undefined) {
				throw "Configuration error - you have to specify dependencies for task '" + taskName;
			}

			taskDependenciesLocalNames.forEach(function(current) {
				taskDependenciesQualifiedNames.push(MODULE_NAME + ':' + current);
			});
			var globalTaskName = MODULE_NAME + ':' + taskName;

			gulp.task(globalTaskName, taskDependenciesQualifiedNames, taskModule.task);
			console.log('Adding ' + globalTaskName + ' to tasks list');
			tasksToRun.push(globalTaskName);
		}
	}

	gulp.task(MODULE_NAME, tasksToRun, function(cb) {
		cb();
	});
};

The createGulpModule function and set of default tasks we use in all our projects (compile, tslint, tsd, sass, test and bundle) were moved to new npm module which we’ve called gulp-commons and put it in our private npm registry. How to run those default tasks in another module? It’s quite straightforward, the whole gulpfile looks like this:

var gulp = require('gulp');
var gulpCommons = require('gulp-commons');

gulpCommons.createGulpModule(gulp,{
	moduleName: "MyAmazingModule",
	cwd: __dirname
});

And it can be run with command gulp MyAmazingModule. Voila!

This approach has these prop:

  • All generic code now lies in a single place – in the gulp-commons module, so no more copypasta!
  • Module specific tasks can be simply added by placing a file in gulp-tasks directory
  • Build definition of each module is now short and clean, you can see what tasks are run be just listing gulp-tasks directory

Moving further – parent modules and submodules

Let’s go bigger with our reusable task definitions. We want to have a parent module which consists of two children moduleA and moduleB. We want to be able to build the whole parent or just single modules. To do that, we will need a short gulpfile in each module and a gulpfile in the parent. In each gulpfile, we will load script gulp-tasks which will call function createGulpModule from gulpCommons. So the directory structure (skipping the source files) will look like this:
.
└── parent
├── gulpfile.js
├── moduleA
│   ├── gulpfile.js
│   └── gulp-tasks.js
└── moduleB
├── gulpfile.js
└── gulp-tasks.js

Gulpfiles in moduleA and moduleB looks exactly the same:

var gulp = require('gulp');
require("./gulp-tasks")(gulp);

In moduleA/gulp-tasks.js we have this piece of code:

module.exports = function(gulp){
	var gulpCommons = require('gulp-commons');

	gulpCommons.createGulpModule(gulp,{
		moduleName: "moduleA",
		cwd: __dirname
	});
}

And moduleB/gulp-tasks.js looks almost the same, except the module name:

module.exports = function(gulp){
	var gulpCommons = require('gulp-commons');

	gulpCommons.createGulpModule(gulp,{
		moduleName: "moduleB",
		cwd: __dirname
	});
}

And the last piece which joins the whole thing together, the gulpfile in parent:

var gulp = require('gulp');
require("./moduleA/gulp-tasks")(gulp);
require("./moduleB/gulp-tasks")(gulp);

gulp.task('default', ['moduleA',''moduleB']);

All modules can be now built with single command – gulp. Or single modules can be built by running command gulp <moduleName> either from parent directory or from its subdirectories. Pretty nice, isn’t it? The idea can be further extended – you can add any level of nesting.

Watch tasks

Gulp has a nice feature which is very handy during development – watch task. It observes a given set of files for changes. When a change happens, it triggers the execution of a task function. We will extend our createGulpModule function to create watch tasks. You might have already noticed that in our tasks definitions we have the fields isWatchTaskwatchTriggersDependencies and srcFiles. The first one is self-describing, but some tasks like running tsd just don’t make sense to be run in watch mode. The last one is self-describing too, srcFiles points to the files going to be observed by the watch task.
What the heck is watchTriggersDependencies?
Each task has dependencies. For example, the typescript compiler depends on tsd which will download typings and on tslint which validates the code. So we have a dependency chain that ends on the task ‘clear’ which, as the name suggests, clears the built directory so that each build process has a fresh workspace. Imagine that we have a watch task for a compiler with enabled triggering dependencies. The first change in the typescript file would trigger clearing the whole built directory. This is definitely not what we wanted. Let’s take a look at the extension we add to createGulpModule:

...
var watchTasksToRun = [];
for (var i = 0; i &amp;amp;amp;amp;amp;amp;amp;amp;lt; taskFiles.length; i++) {
	...
	if (taskModule.isWatchTask){
		var watchDependencies = [];
		if (taskModule.watchTriggersDependencies){
			taskDependenciesQualifiedNames.forEach(function(current) {
				watchDependencies.push(current + ":watch");
			});
		}
		watchTasksToRun.push({
			name: globalTaskName + ":watch",
			dependencies: watchDependencies,
			srcFiles: taskModule.srcFiles(),
			task: taskModule.task
		}
		);
	}

}

gulp.task(MODULE_NAME + ":watch", function(){
	for (var i=0; i&amp;amp;amp;amp;amp;amp;amp;amp;lt; watchTasksToRun.length; i++){
		var watchTask = watchTasksToRun[i];
		gulp.task(watchTask.name, watchTask.dependencies, watchTask.task);
		gulp.watch(watchTask.srcFiles, {cwd:options.cwd}, [watchTask.name]);
	}
});

Now running gulp myAmazingModule:watch will start a watching task.

Source code

Full source code of examples used in the article is available on GitHub here.

Final thoughts

I hope tips from this article will be useful for others. Node js is young (or even immature IMHO) technology. Building and testing tools still lack the stability and seem premature when compared to the toolset found in Java. However, things are changing fast and there is a great community supporting the project. In the next article, I will show some tricks using npm and local packages linking. See you next time!

comments: 0


Notice: Theme without comments.php is deprecated since version 3.0.0 with no alternative available. Please include a comments.php template in your theme. in /var/www/html/www_en/wp-includes/functions.php on line 3937

Leave a Reply

Your email address will not be published. Required fields are marked *