Using external node dependencies
This guide will teach you how to setup a project with Webpack to bundle your Phantombuster script.
Even though the Phantombuster packages offer you a wide range of modules for your custom scripts, you might want to add additional node packages that are not available directly in the Phantombuster packages.
Before reading this guide
Before diving deeper into this guide, make sure you read our previous guides on managing your custom scripts and coding a custom script.
Using a bundler
A bundler is a tool which takes all your code, dependencies and assets and bundles them together for distribution. It also allows you to add post-processing to your code to minify or obfuscate it.
Using a bundler will allow you to bundle your node dependencies directly with your script code, thus meaning we can use any node dependency even if it is not included in a Phantombuster package.
For demonstration purposes we will use Webpack to show you how to bundle your code.
Its main purpose is to bundle JavaScript files for usage in a browser, but it is also capable of bundling and packaging node applications.
API limitation
Phantombuster has a 600kb limit on script upload. Using a bundler with too many dependencies might cause your script to exceed this limit.
Check the alternative method to circumvent this issue.
Setting up the project
The first step is to install webpack and the webpack-cli:
npm install webpack webpack-cli --save-dev
For the purposes of the guide we will install the colors
npm package - a very popular module to get colors in the node.js console.
Note that this module is not included in the Phantombuster packages.
npm install colors --save
Now you can write the code for your script.
Let's create a simple Phantom which saves some text using Buster.saveText
and displays a text in the console with your previously installed colors
package.
Let's start by creating an index.js
file in the src
folder:
mkdir src
touch src/index.js
Now write the following code in the newly created index.js
file:
"phantom image: web-node:v1"
const colors = require('colors/safe');
const Buster = require("phantombuster");
const buster = new Buster();
async function main() {
await buster.saveText("some random text", "test.txt");
console.log(colors.green('Hello from Phantombuster'));
}
main();
Your project is now almost ready to be bundled, we only need to create a configuration file for Webpack in order for it to properly bundle your code and make it work with Phantombuster.
Create the webpack.config.js
file in the root of your project:
const path = require('path');
module.exports = {
target: "node", // this will tell webpack to compile for a Node.js environment
entry: {
app: ["./src/index.js"]
},
output: {
path: path.resolve(__dirname, "./build"),
filename: "bundle.js"
},
externals: {
phantombuster: "commonjs2 phantombuster", // this makes sure webpack doesn't try to bundle the "phantombuster" module
},
optimization:{
minimize: false, // we disable the minimization plugin to keep the phantombuster package comment directives
},
};
Your code is now ready to be bundled!
You can change the output
and entry
properties of the configuration to fit your needs.
This configuration will output the bundled code to ./build/bundle.js
.
Bundling the code
Now that our project is set up, we need to bundle it before uploading it to Phantombuster.
Run the following command from the root of your project:
npx webpack
If you installed Webpack globally you can just run:
webpack
As you can see, Webpack created a build
folder containing a new bundle.js
file.
If you open it up, you will notice that a lot of additional code has been added. This is because Webpack resolved all the dependencies from your code (eg: the colors
module) and included them directly in the source file.
You can also run Webpack in watch
mode. This will allow you to automatically bundle your code on changes.
To do so, run the following command:
npx webpack --watch
You can now configure the Phantombuster sdk to publish your bundled code :)
Using Phantombuster packages along with custom dependencies
When bundling your code with Webpack, you won't be able to use Phantombuster's package dependencies as they will try to be resolved by Webpack.
If you still want to be able to use some of the Phantombuster packages, you can use the externals
option of the Webpack configuration file.
The externals
configuration option provides a way of excluding dependencies from the output bundles.
Example: If you want to use the puppeteer
module from the Phantombuster package, use the following configuration:
module.exports = {
// ...
externals: {
'phantombuster': 'commonjs2 phantombuster',
'puppeteer': 'commonjs2 puppeteer',
}
// ...
};
Alternatives
If you don't want to setup Webpack with your project, you can use an alternative method to directly install an npm module from your code.
Please note that using with technique will result in more execution time and could slow down your script.
The following example shows you how to install the cowsay
module directly from code:
"phantom image: web-node:v1"
const exec = require("child_process").exec;
exec("npm init --yes", (/* error, stdout, stderr */) => {
exec("npm install cowsay", (/* error, stdout, stderr */) => {
// Make node detect the newly added node_modules folder
process.env.NODE_PATH =
"/home/phantom/node_modules:/home/phantom/agent/node_modules";
require("module").Module._initPaths();
// Test that our new module is indeed available
const cowsay = require("cowsay");
console.log(
cowsay.say({
text: "I'm a moooodule",
e: "oO",
T: "U ",
})
);
});
});
Updated over 1 year ago