Even though the Phantombuster packages offer you a wide range of modules for your custom scripts, you might want to add additional node packages that are not available directly in the Phantombuster packages.

📘

Before reading this guide

Before diving deeper into this guide, make sure you read our previous guides on managing your custom scripts and coding a custom script.

Using a bundler

A bundler is a tool which takes all your code, dependencies and assets and bundles them together for distribution. It also allows you to add post-processing to your code to minify or obfuscate it.

Using a bundler will allow you to bundle your node dependencies directly with your script code, thus meaning we can use any node dependency even if it is not included in a Phantombuster package.

For demonstration purposes we will use Webpack to show you how to bundle your code.
Its main purpose is to bundle JavaScript files for usage in a browser, but it is also capable of bundling and packaging node applications.

🚧

API limitation

Phantombuster has a 600kb limit on script upload. Using a bundler with too many dependencies might cause your script to exceed this limit.
Check the alternative method to circumvent this issue.

Setting up the project

The first step is to install webpack and the webpack-cli:

npm install webpack webpack-cli --save-dev

For the purposes of the guide we will install the colors npm package - a very popular module to get colors in the node.js console.
Note that this module is not included in the Phantombuster packages.

npm install colors --save

Now you can write the code for your script.
Let's create a simple Phantom which saves some text using Buster.saveText and displays a text in the console with your previously installed colors package.

Let's start by creating an index.js file in the src folder:

mkdir src
touch src/index.js

Now write the following code in the newly created index.js file:

"phantom image: web-node:v1"

const colors = require('colors/safe');
const Buster = require("phantombuster");

const buster = new Buster();

async function main() {
	await buster.saveText("some random text", "test.txt");
	console.log(colors.green('Hello from Phantombuster'));
}

main();

Your project is now almost ready to be bundled, we only need to create a configuration file for Webpack in order for it to properly bundle your code and make it work with Phantombuster.

Create the webpack.config.js file in the root of your project:

const  path = require('path');

module.exports = {
	target:  "node", // this will tell webpack to compile for a Node.js environment 
	entry: {
		app: ["./src/index.js"]
	},
	output: {
		path:  path.resolve(__dirname, "./build"),
		filename:  "bundle.js"
	},
	externals: {
		phantombuster:  "commonjs2 phantombuster", // this makes sure webpack doesn't try to bundle the "phantombuster" module
	},
	optimization:{
		minimize:  false, // we disable the minimization plugin to keep the phantombuster package comment directives
	},
};

Your code is now ready to be bundled!
You can change the output and entry properties of the configuration to fit your needs.
This configuration will output the bundled code to ./build/bundle.js.

Bundling the code

Now that our project is set up, we need to bundle it before uploading it to Phantombuster.

Run the following command from the root of your project:

npx webpack

If you installed Webpack globally you can just run:

webpack

As you can see, Webpack created a build folder containing a new bundle.js file.
If you open it up, you will notice that a lot of additional code has been added. This is because Webpack resolved all the dependencies from your code (eg: the colors module) and included them directly in the source file.

You can also run Webpack in watch mode. This will allow you to automatically bundle your code on changes.
To do so, run the following command:

npx webpack --watch

You can now configure the Phantombuster sdk to publish your bundled code :)

Using Phantombuster packages along with custom dependencies

When bundling your code with Webpack, you won't be able to use Phantombuster's package dependencies as they will try to be resolved by Webpack.

If you still want to be able to use some of the Phantombuster packages, you can use the externals option of the Webpack configuration file.
The externals configuration option provides a way of excluding dependencies from the output bundles.

Example: If you want to use the puppeteer module from the Phantombuster package, use the following configuration:

module.exports = {
  // ...
  externals: {
    'phantombuster': 'commonjs2 phantombuster',
    'puppeteer': 'commonjs2 puppeteer',
  }
  // ...
};

Alternatives

If you don't want to setup Webpack with your project, you can use an alternative method to directly install an npm module from your code.
Please note that using with technique will result in more execution time and could slow down your script.

The following example shows you how to install the cowsay module directly from code:

"phantom image: web-node:v1"

const exec = require("child_process").exec;

exec("npm init --yes", (/* error, stdout, stderr */) => {
  exec("npm install cowsay", (/* error, stdout, stderr */) => {
    // Make node detect the newly added node_modules folder
    process.env.NODE_PATH =
      "/home/phantom/node_modules:/home/phantom/agent/node_modules";
    require("module").Module._initPaths();

    // Test that our new module is indeed available
    const cowsay = require("cowsay");
    console.log(
      cowsay.say({
        text: "I'm a moooodule",
        e: "oO",
        T: "U ",
      })
    );
  });
});