We Migrated the Tooljet Server From Ruby to Node.js

In recent, JavaScript has become the common language for both backend and frontend applications. The complete thought process behind porting the ToolJet server from Ruby to Node.js is explained in our blog post. It took us more than 3 weeks to port the server completely to Node.js. We first spent a few days researching the frameworks to use. We decided to use NestJS with TypeORM as the ORM.

For those who haven't heard about ToolJet, ToolJet (https://github.com/ToolJet/ToolJet) is an open-source low-code framework for building custom internal tools.

Our focus was to make the migration seamless for our users, it meant that there should not be any effort on our users' part. This meant that the data in users' databases need not be modified to use the newer versions of ToolJet.

ToolJet can connect to a bunch of data sources such as databases, API endpoints, and external services. The credentials for these data sources are encrypted and stored in a Postgres database. We were using Lockbox to encrypt and decrypt the credentials securely. Lockbox uses AES-GCM as the default algorithm. Lockbox uses a master key and generates derived keys for every column that needs to be encrypted.

We had to go through the codebase of Lockbox to implement the same logic of deriving keys based on a maser key in Node.js, otherwise, our users will have to update every credential stored in the database manually. We used futoin-hkdf npm package to generate derived keys based on the table name and column name using hkdf ( Note: if you are using node v16, crypto.hkdf can be used instead ).

computeAttributeKey(table: string, column: string): string {    const key = Buffer.from(process.env.LOCKBOX_MASTER_KEY, 'hex');    const salt = Buffer.alloc(32, '´', 'ascii');    const info = Buffer.concat([salt, Buffer.from(`${column}_ciphertext`)]);    const derivedKey = hkdf(key, 32, {salt: table, info, hash: 'sha384'});    const finalDerivedKey = Buffer.from(derivedKey).toString('hex');    return finalDerivedKey;  }

This part was very easy. We just had to use @nestjs/jwt package and bcrypt to let the users continue using their login credentials.

ToolJet had integrations with multiple databases, API-based services, etc at the time of migration. To migrate to Node.js, we just had to find a suitable Node.js library for each of the data sources and rewrite the code for querying the data sources.

The existing users will be able to use their existing databases without any changes with the new version of ToolJet. But for new users, we had to build TypeORM migrations to create the database tables. TypeORM doesn't have a native way to create or drop databases, we had to write our own scripts.

NestJS provides Jest and Supertest integrations out of the box. We had to make some changes to reset the database before running the test suite and to clear the tables before running each test. The details of the implementation are documented here.

ToolJet needs to send emails such as invitation emails for new users. We were using ActionMailer to send emails from the Ruby on Rails application. We decided to use nodemailer to send emails from the Node.js server. It was easy to set up and works seamlessly. In the development environment, it makes sense to preview the emails on the browser. We used preview-email npm package to implement email previews.

Once the codebase was migrated, we had to update our setup scripts. Since ToolJet is commonly used as a self-hosted tool, we support multiple deployment options such as Kubernetes, Heroku, EC2, etc. This part was easy and straightforward as we already had Node.js in our setup scripts to build and run the Tooljet client.

We took the migration as an opportunity to refactor code and to improve the test coverage of ToolJet. Our next goal is to make it easier for developers to build ToolJet integrations.

We would love you to check out ToolJet on GitHub: https://github.com/ToolJet/ToolJet/