Writing custom API agents

Important general information on how to create and configure custom API agents on Phantombuster

At its core, Phantombuster allows you to script "web robots" in JavaScript. We call them API agents because they're bots that each have an HTTP API endpoint.

API agents can come from the API store or be totally custom and created by you. Creating and configuring custom API agents is what we'll explain in this page.

How to create a new custom API agent

To create a new custom API agent, log in, go to your APIs page and simply enter a name. The "Template" option will initialize your agent with a script. If you're not sure, select "NickJS".

Script directives

All Phantombuster scripts must start with a few lines called script directives. They let you configure the environment in which the script will run.

Here is a typical example:

// Phantombuster configuration {
"phantombuster package: 5"
"phantombuster command: nodejs"
"phantombuster dependencies: lib-Foo.js"
"phantombuster flags: save-folder"
// }

Each line is a directive and will be explained below. Their order is of no importance.

Auto folding "Phantombuster configuration" block

When you write // Phantombuster configuration { and add the corresponding // }, our code editor will automatically fold the block so that the beginning of your script becomes more readable.

After all, what's important is the code you write, not the script directives :)

Packages ("phantombuster package")

The phantombuster package is one of the most important script directives. It allows you to select which NPM modules and binaries are bundled with your agent's script. It is mandatory to specify a package.

"phantombuster package: 5"

A list of all the available packages and modules here:
Packages & modules →

Binaries ("phantombuster command")

Scripts can be launched on our platform by one of the following binaries:

  • NodeJS — Execute your scripts in V8, Chrome's JavaScript runtime — Allows you to control Headless Chrome
  • PhantomJS — Headless, scriptable WebKit browser (where Phantombuster originally got its name from) — Now DEPRECATED, please use NodeJS instead
  • CasperJS — Framework built on top of PhantomJS to write test and navigation scenarios — Now DEPRECATED, please use NodeJS instead

Telling your script which binary to use

Use the phantombuster command directive to select the binary you want, for example:

"phantombuster command: nodejs"

The available keywords are: nodejs, phantomjs and casperjs.

Always specify a binary

If you omit the phantombuster command directive, Phantombuster will launch CasperJS by default. Always specify which binary to use!

Writing your own modules ("phantombuster dependencies")

Sometimes your custom API agents become big and the size of their scripts make them difficult to understand and maintain.

That's the time when you need to split your code into multiple files. Phantombuster allows you to create modules that you can require() into your agent's code.

How to create a new module

To create a new module, log in, go to your modules page, enter the desired name of your module and click "Create" (module names are automatically prefixed with lib-).

Example of a simple module:

module.exports = {
  foo: () => {
  	console.log("bar");
  }
}

Example of an agent script using the module shown above:

"phantombuster command: node"
"phantombuster package: 5"
"phantombuster dependencies: lib-Foo.js, lib-OtherModule.js"

require("./lib-Foo").foo() // outputs "bar"

As you can see, you must indicate on which modules your agent's script is dependent before being able to require() them.

When calling require(), do not forget to add ./ in front of the name of your module. If your agent's script has multiple dependencies, specify a list of module names separated by commas.

The dependency list is case sensitive and must contain the exact modules names including their extension.

Flags ("phantombuster flags")

The phantombuster flags directive allows you to enable miscellaneous features of the Phantombuster platform.

For now there is only one flag named save-folder which makes sure the files you have downloaded to your agent's disk get saved to your account's cloud storage before the script terminates.

"phantombuster flags: save-folder"

This lets you write your scripts without bothering to remember to call buster.saveFolder() at the end.

In what environment do my scripts run?

Your scripts are executed in Linux containers (they are similar to very light virtual machines).

Available to you are a few gigabytes of RAM, a few gigabytes of hard disk space and a fast internet connection. These are temporary resources that are freed right after your agent finishes its job.

What's important to know is that files written on your agent's disk will be lost when it exits. To keep files, save them to your persistent storage using our BusterJS agent module or by adding the line "phantombuster flags: save-folder" at the top of your script.

More technical details (for the nerds):

  • The container engine is Docker
  • Containers are running Debian
  • Agents always start in /home/phantom/agent which is empty
  • Agents run under the user phantom

Phantombuster's SDK

All your scripts can easily be written right on our website, in the provided JavaScript web editor.

However, you might prefer using your own editor, locally on your machine. We made Phantombuster's SDK specifically for this.

The SDK will monitor a directory on your disk for changes in your scripts. As soon as a change is detected, the script will be uploaded in your Phantombuster account.

First, you need to have npm installed. Then do this:

# npm install -g phantombuster-sdk

It will globally install the phantombuster command. Discover how to use it →