Log in to a website
A simple guide to understand how to log in to any website with Phantombuster.
As you know a lot of websites have their own identification system. Behind this identification, there is often highly valuable data for your business or personal use. It could also be data you generated with your personal account on the target site, and now you want to get it out to integrate it somewhere else.
Most of the scrapers out there don't handle this kind of stuff. But don't worry, with Phantombuster it's easy. Let's see!
Let's fill a login form
First of all, how do you log in to an account on a website?
When you do it manually as a human, you click on each input and fill them one by one, then you click to submit the form. That's exactly what you can do with Phantombuster and the following Puppeteer's methods:
await page.type("#email", "[email protected]")
await page.type("#password", "johnjohn")
await page.click("button[type=\"submit\"]")
Whoa... so many things? I am going to explain all of this.
- The first argument of both these methods is a CSS selector, you can find it using your developer console on the website you want to scrape. Here the login form looks like this:
<form method="post" action="/login/auth">
<div class="form-group">
<label for="email">Email</label>
<input type="email" class="form-control" name="email" id="email">
</div>
<div class="form-group">
<label for="password">Password</label>
<input type="password" class="form-control" name="password" id="password">
</div>
<button type="submit" class="btn btn-default">Connect</button>
</form>
We can see that the form contains two fields (html <input/>
) having id
attributes. The CSS selector for an id is #<the_id>
. In our case: #email
and #password
!
- We fill both these fields with our credentials.
- The form is ready to be submited, we just have to click the submit button! The button's selector is
button[type="submit"]
, meaning abutton tag having a type attribute of value "submit"
.
Complete example
For this example, I am going to use our challenges site.
You'll need to log in with these credentials:
[email protected]
as email and johnjohn
as the password.
- First create new a custom script.
- You can now edit your script in our online editor or using our SDK.
"phantombuster command: nodejs"
"phantombuster package: 5"
"phantombuster flags: save-folder"
/**
* The upper part is phantombuster configuration
* Check out our guide on writing custom scripts
* @see: https://hub.phantombuster.com/docs/script-directives
* If you would like to know more about script directives
*/
const Buster = require("phantombuster")
const puppeteer = require("puppeteer")
const buster = new Buster()
const scrapePreciousData = () => {
// we are in the browser's scope there
const data = []
document.querySelectorAll("div.person > div.panel-body").forEach((person) => {
data.push({
name: person.querySelector(".name").innerText.trim(),
birthYear: person.querySelector(".birth_year").innerText.trim(),
gender: person.querySelector(".gender").innerText.trim(),
job: person.querySelector(".job").innerText.trim(),
})
})
return data
}
;(async () => {
const browser = await puppeteer.launch({
// This is needed to run Puppeteer in a Phantombuster container
args: ["--no-sandbox"]
})
// Go to the page and wait for it to load
const page = await browser.newPage()
await page.goto("http://scraping-challenges.phantombuster.com/login")
await page.waitForSelector("form")
// fill and submit the login form
await page.type("#email", "[email protected]")
await page.type("#password", "johnjohn")
await page.click("button[type=\"submit\"]")
// wait for the "private" data to appear!
await page.waitForSelector(".panel-body")
// take a screenshot to make sure everything is fine
await page.screenshot({ path: "screenshot.png", fullPage: true })
// scrape all the precious data
const preciousData = await page.evaluate(scrapePreciousData)
// and save it to your result object
await buster.setResultObject(preciousData)
// clean up everything
await page.close()
await browser.close()
process.exit()
})()
References:
- Puppeteer.launch
- Puppeteer.Browser.newPage
- Puppeteer.Page.goto
- Puppeteer.Page.waitForSelector
- Puppeteer.Page.type
- Puppeteer.Page.click
- Puppeteer.Page.screenshot
- Puppeteer.Page.evaluate
- Buster.setResultObject
- Puppeteer.Page.close
- Puppeteer.Browser.close
Watch your agent do its job
Launch it and enjoy your data!

Retrieve the data
π‘We used buster.setResultObject to store the result object. We can now retrieve it using Phantombuster's API.
Updated over 2 years ago