Scraping Browser Troubleshooting

Scraping Browser Debugger

Web scraping projects often require intricate interactions with target websites and debugging is vital for identifying and resolving issues found during the development process. The Scraping Browser Debugger serves as a valuable resource, enabling you to inspect, analyze, and fine-tune your code alongside Chrome Dev Tools, resulting in better control, visibility, and efficiency.

Where to find it

Our Scraping Browser Debugger can be launched via two methods:

Manually via Control Panel
Remotely via your script

Choose your preferred method below to see more:

Manually via Control Panel
Remotely via your script

The Scraping Browser Debugger can be easily accessed within your Bright Data Control Panel. Follow these steps:

Within the control panel, go to My Zones
Click on your specific Scraping Browser zone
Click on the Overview tab
On the right side, Click on the “Chrome Dev Tools Debugger” button seen below:

Getting Started with the Debugger & Chrome Dev Tools

Open a Scraping Browser Session

Ensure you have an active Scraping Browser session
If you don’t yet know how to launch a scraping browser session, see our Quick Start guide.

Launch the Debugger

Once your session is up and running you can now launch the Debugger.
Click on the Debugger button within your ‘Access parameters’ tab to launch the Scraping Browser Debugger interface (see the screenshot above )

Connect with your live browser sessions

Within the Debugger interface, you will find a list of your live Scraping Browser sessions.
Select the preferred session that you wish to debug
Click on the session link or copy/paste it into your browser of choice, and this will establish a connection between the Debugger and your selected session.

To access and launch the debugger session directly from your script, you’ll need to send the CDP command: Page.inspect.See the following examples:

// Puppeteer - Inspect page using devtools
const page = await browser.newPage();
const client = await page.target().createCDPSession();
const {frameTree: {frame}} = await client.send('Page.getFrameTree', {});
const {url: inspectUrl} = await client.send('Page.inspect', {
    frameId: frame.id,
});
console.log(`Inspect session at ${inspectUrl}`);

// Playwright - Inspect page using devtools
const page = await browser.newPage();
const client = await page.context().newCDPSession(page);
const {frameTree: {frame}} = await client.send('Page.getFrameTree', {});
const {url: inspectUrl} = await client.send('Page.inspect', {
    frameId: frame.id,
});
console.log(`Inspect session at ${inspectUrl}`);

# Playwright - Inspect page using devtools
page = await browser.new_page()
client = await page.context.new_cdp_session(page)
frame_tree = await client.send('Page.getFrameTree', {})
frame_id = frame_tree['frameTree']['frame']['id']
inspect = await client.send('Page.inspect', {'frameId': frame_id})
inspect_url = inspect['url']
print('Inspect session at', inspect_url)

// PuppeteerSharp - Inspect page using devtools
var page = await browser.NewPageAsync();
var client = await page.Target.CreateCDPSessionAsync();
var frameTree = await client.SendAsync("Page.getFrameTree");
var frameId = frameTree!["frameTree"]!["frame"]!["id"]!;
var inspect = await client.SendAsync("Page.inspect", new { frameId = frameId });
var inspectUrl = inspect["url"]!;
Console.WriteLine($"Inspect session at {inspectUrl}");

// Playwright - Inspect page using devtools
var page = await browser.NewPageAsync();
var client = await page.Context.NewCDPSessionAsync(page);
var frameTree = await client.SendAsync("Page.getFrameTree");
var frameId = frameTree.Value
    .GetProperty("frameTree")
    .GetProperty("frame")
    .GetProperty("id")
    .GetString()!;
var inspect = await client.SendAsync("Page.inspect", new ()
{
    { "frameId", frameId },
});
var inspectUrl = inspect.Value
    .GetProperty("url")
    .GetString()!;
Console.WriteLine($"Inspect session at {inspectUrl}");

Leveraging Chrome Dev Tools

With the Scraping Browser Debugger now connected to your live session, you gain access to the powerful features of Chrome Dev Tools.
Utilize the Dev Tools interface to inspect HTML elements, analyze network requests, debug JavaScript code, and monitor performance. Leverage breakpoints, console logging, and other debugging techniques to identify and resolve issues within your code.

Automatically opening devtools locally to view your live browser session

If you would like to automatically launch devtools on every session to view your live browser session, you can integrate the following code snippet:

NodeJS - Puppeteer

// Launch devtools locally

const { exec } = require('child_process');
const chromeExecutable = 'google-chrome';

const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
const openDevtools = async (page, client) => {
    // get current frameId
    const frameId = page.mainFrame()._id;
    // get URL for devtools from scraping browser
    const { url: inspectUrl } = await client.send('Page.inspect', { frameId });
    // open devtools URL in local chrome
    exec(`"${chromeExecutable}" "${inspectUrl}"`, error => {
        if (error)
            throw new Error('Unable to open devtools: ' + error);
    });
    // wait for devtools ui to load
    await delay(5000);
};

const page = await browser.newPage();
const client = await page.target().createCDPSession();
await openDevtools(page, client);
await page.goto('http://example.com');

Debugger Walkthrough

Check out the Scraping Browser Debugger in action below

Error codes

Analyze some of the popular error codes you might be receiving below:

Error Code	Meaning	What can you do about it?
`407`	An issue with the remote browser’s port	Please check your remote browser’s port. The correct port for Scraping Browser is either the default port:9222 OR if you are using Selenium specifically, use port:9515
`403`	Authentication Error	Check authentication credentials (username, password) and check that you are using the correct “Browser API” zone from Bright Data control panel
`503`	Service Unavailable	We are likely scaling browsers right now to meet demand. Try to reconnect in 1 minute.

FAQs

Check out our frequently asked questions regarding Scraping Browser

How To

Find out more about the common library navigational functions and examples for browser automation and specifically for Scraping Browser

​Scraping Browser Debugger

​Where to find it

​Getting Started with the Debugger & Chrome Dev Tools

​Leveraging Chrome Dev Tools

​Automatically opening devtools locally to view your live browser session

​Debugger Walkthrough

​Error codes

​FAQs

​How To