> ## Documentation Index
> Fetch the complete documentation index at: https://brightdata-ipv6-release.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Trigger data collection API

> Learn how to trigger data collection using the Web Scraper API with options for discovery and PDP scrapers. Customize requests, set delivery options, and retrieve data efficiently.

## How It Works

By default, scraping requests are processed asynchronously. When a request is submitted, the system begins processing the job in the background and immediately returns a snapshot ID. Once the scraping task is complete, the results can be retrieved at your convenience by using the snapshot ID to download the data via the API. Alternatively, you can configure the request to automatically deliver the results to an external storage destination, such as an S3 bucket or Azure Blob Storage. This approach is well-suited for handling larger jobs or integrating with automated data pipelines.

## Body

The inputs to be used by the scraper. Can be provided either as JSON or as a CSV file:

<ParamField body="Content-Type" type="string">
  A JSON array of inputs

  > **Example**: `[{"url":"https://www.airbnb.com/rooms/50122531"}]`

  ***

  A CSV file, in a field called `data`

  > **Example** (curl): `data=@path/to/your/file.csv`
</ParamField>

## Web Scraper Types

Each scraper can require different inputs. There are 2 main types of scrapers:

### 1. PDP

These scrapers require URLs as inputs. A PDP scraper extracts detailed product information like specifications, pricing, and features from web pages

### 2. Discovery

Discovery scrapers allow you to explore and find new entities/products through search, categories, Keywords and more.

<Frame>
  <img src="https://mintcdn.com/brightdata-ipv6-release/yAA1dELwzqgxV65_/images/scraping-automation/web-scraper-api/ae.com.png?fit=max&auto=format&n=yAA1dELwzqgxV65_&q=85&s=dac882a084510e3e7d543525687ddea5" alt="ae.com.png" width="722" height="162" data-path="images/scraping-automation/web-scraper-api/ae.com.png" />
</Frame>

## Request examples

### `PDP` with URL input

Input format for `PDP` is always a URL, pointing to the page to be scraped.

```sh Sample Request theme={null}
curl -H "Authorization: Bearer API_TOKEN" -H "Content-Type: application/json" -d '[{"url":"https://www.airbnb.com/rooms/50122531"},{"url":"https://www.airbnb.com/rooms/50127677"}]' "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_ld7ll037kqy322v05&format=json&uncompressed_webhook=true"
```

### Discovery input based on the `discovery` method

```sh Sample Request theme={null}
curl -H "Authorization: Bearer x2x3fdaaddrer" -H "Content-Type: application/json" -d '[{"keyword":"light bulb"},{"keyword":"dog toys"},{"keyword":"home decor"}]' "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l7q7dkf244hwjntr0&endpoint=https://webhook-url.com&auth_header=QWxhZGRpbjpPcGVuU2VzYW1l&notify=https://notify-me.com/&format=ndjson&uncompressed_webhook=true&type=discover_new&discover_by=keyword&limit_per_input=10"
```

Input format for `discovery` can vary according to the specific scraper. Inputs can be:

<CodeGroup>
  ```JSON keywords theme={null}
  [{"keyword": "light bulb"},{"keyword": "dog toys"},{"keyword": "home decor"}]
  ```

  ```JSON Search settings theme={null}
  [{"url": "https://www.amazon.com/s?i=luggage-intl-ship", "sort_by": "Best Sellers"}]
  ```

  ```JSON Locations theme={null}
  [{"location": "Europe"},{"location": "Greece"},{"location": "United States"}]
  ```
</CodeGroup>

And more. Find out what inputs each scraper requires [here](https://brightdata.com/cp/data_api).


## OpenAPI

````yaml dca-api POST /datasets/v3/trigger
openapi: 3.1.0
info:
  title: Brightdata API
  description: API for interaction with datasets marketplace
  version: 1.0.0
servers:
  - url: https://api.brightdata.com
security:
  - bearerAuth: []
paths:
  /datasets/v3/trigger:
    post:
      description: Scrape the target website based on the inputs provided in body
      parameters:
        - name: dataset_id
          description: Dataset ID for which data collection is triggered.
          in: query
          required: true
          schema:
            type: string
            example: gd_l1vikfnt1wgvvqz95w
        - name: type
          in: query
          schema:
            type: string
            enum:
              - discover_new
          description: >-
            Set it to "discover_new" to trigger a collection that includes a
            discovery phase.
        - name: discover_by
          in: query
          schema:
            type: string
          description: >-
            Specifies which discovery method to use. Available options:
            "keyword", "best_sellers_url", "category_url", "location" and more
            (according to the specific API). Relevant only for collections that
            include a discovery phase.
        - name: include_errors
          in: query
          schema:
            type: boolean
          description: Include errors report with the results.
        - name: limit_per_input
          in: query
          schema:
            type: number
            minimum: 1
          description: >-
            Limit the number of results per input. Relevant only for collections
            that include a discovery phase.
        - name: limit_multiple_results
          in: query
          schema:
            type: number
            minimum: 1
          description: Limit the total number of results.
        - name: notify
          in: query
          schema:
            type: string
          description: >-
            URL where the notification will be sent once the collection is
            finished. Notification will contain snapshot_id and status.
        - name: endpoint
          in: query
          schema:
            type: string
          description: Webhook URL where data will be delivered.
        - name: format
          in: query
          schema:
            type: string
            enum:
              - json
              - ndjson
              - jsonl
              - csv
          description: >-
            Specifies the format of the data to be delivered to the webhook
            endpoint.
        - name: auth_header
          in: query
          schema:
            type: string
          description: >-
            Authorization header to be used when sending notification to notify
            URL or delivering data via webhook endpoint.
        - name: uncompressed_webhook
          in: query
          schema:
            type: boolean
          description: >-
            By default, the data will be sent to the webhook compressed. Pass
            true to send it uncompressed.
      requestBody:
        required: true
        content:
          application/json:
            schema:
              anyOf:
                - $ref: '#/components/schemas/TriggerInput'
                  title: Only inputs
                - $ref: '#/components/schemas/TriggerAndDeliverBody'
                  title: Deliver config and inputs
          multipart/form-data:
            schema:
              type: string
              description: A CSV file, in a field called data
      responses:
        '200':
          description: Collection job successfully started
          content:
            application/json:
              schema:
                type: object
                properties:
                  snapshot_id:
                    type: string
                    description: ID of your request that can be used in the next APIs
                    example: s_m4x7enmven8djfqak
components:
  schemas:
    TriggerInput:
      type: array
      example:
        - url: https://il.linkedin.com/company/bright-data
      items:
        type: object
        additionalProperties: {}
    TriggerAndDeliverBody:
      type: object
      example:
        deliver:
          type: webhook
          filename:
            extension: json
            template: '{[snapshot_timestamp]}_{[snapshot_id]}'
          endpoint: https://example.com/foo/bar
        input:
          - url: https://il.linkedin.com/company/bright-data
      properties:
        deliver:
          $ref: '#/components/schemas/DeliverConfig'
        input:
          $ref: '#/components/schemas/TriggerInput'
    DeliverConfig:
      description: Deliver configuration
      oneOf:
        - $ref: '#/components/schemas/DeliverConfigWebhook'
          title: Webhook
        - $ref: '#/components/schemas/DeliverConfigGCS'
          title: Google Cloud
        - $ref: '#/components/schemas/DeliverConfigGCSPubSub'
          title: Google Cloud PubSub
        - $ref: '#/components/schemas/DeliverConfigS3'
          title: Amazon S3
        - $ref: '#/components/schemas/DeliverConfigSnowflake'
          title: Snowflake
        - $ref: '#/components/schemas/DeliverConfigAliOSS'
          title: Aliyun Object Storage Service
        - $ref: '#/components/schemas/DeliverConfigSFTP'
          title: SFTP
        - $ref: '#/components/schemas/DeliverConfigAzure'
          title: Microsoft Azure
    DeliverConfigWebhook:
      allOf:
        - $ref: '#/components/schemas/DeliverConfigBase'
        - type: object
          properties:
            type:
              enum:
                - webhook
            endpoint:
              type: string
              format: uri
              description: The endpoint URL for the webhook.
    DeliverConfigGCS:
      allOf:
        - $ref: '#/components/schemas/DeliverConfigBase'
        - type: object
          properties:
            type:
              enum:
                - gcs
            bucket:
              type: string
              description: Name of the bucket.
            credentials:
              type: object
              additionalProperties: false
              description: Credentials for authentication
              properties:
                client_email:
                  type: string
                private_key:
                  type: string
              required:
                - client_email
                - private_key
            directory:
              type: string
              description: Target path
          required:
            - bucket
            - credentials
    DeliverConfigGCSPubSub:
      allOf:
        - $ref: '#/components/schemas/DeliverConfigBase'
        - type: object
          properties:
            type:
              enum:
                - gcs_pubsub
            topic_id:
              type: string
            attributes:
              type: array
              items:
                type: object
            credentials:
              type: object
              additionalProperties: false
              properties:
                client_email:
                  type: string
                private_key:
                  type: string
              required:
                - client_email
                - private_key
          required:
            - topic_id
            - credentials
    DeliverConfigS3:
      allOf:
        - $ref: '#/components/schemas/DeliverConfigBase'
        - type: object
          properties:
            type:
              enum:
                - s3
            bucket:
              type: string
            credentials:
              type: object
              additionalProperties: false
              minProperties: 2
              properties:
                aws-access-key:
                  type: string
                aws-secret-key:
                  type: string
                role_arn:
                  type: string
                external_id:
                  type: string
              oneOf:
                - title: Role ARN
                  required:
                    - role_arn
                    - external_id
                - title: Access Key
                  required:
                    - aws-access-key
                    - aws-secret-key
            region:
              type: string
            directory:
              type: string
          required:
            - bucket
            - credentials
    DeliverConfigSnowflake:
      allOf:
        - $ref: '#/components/schemas/DeliverConfigBase'
        - type: object
          properties:
            type:
              enum:
                - snowflake
            database:
              type: string
            schema:
              type: string
            stage:
              type: string
            role:
              type: string
            warehouse:
              type: string
            credentials:
              type: object
              additionalProperties: false
              properties:
                account:
                  type: string
                user:
                  type: string
                password:
                  type: string
              required:
                - account
                - user
                - password
          required:
            - database
            - schema
            - stage
            - role
            - warehouse
            - credentials
    DeliverConfigAliOSS:
      allOf:
        - $ref: '#/components/schemas/DeliverConfigBase'
        - type: object
          properties:
            type:
              enum:
                - ali_oss
            bucket:
              type: string
            credentials:
              type: object
              additionalProperties: false
              properties:
                access-key:
                  type: string
                secret-key:
                  type: string
              required:
                - access-key
                - secret-key
            region:
              type: string
            directory:
              type: string
          required:
            - bucket
            - credentials
            - region
    DeliverConfigSFTP:
      allOf:
        - $ref: '#/components/schemas/DeliverConfigBase'
        - type: object
          properties:
            type:
              enum:
                - sftp
            path:
              type: string
              format: hostname
            port:
              type: integer
              minimum: 0
              maximum: 65535
            credentials:
              type: object
              additionalProperties: false
              properties:
                username:
                  type: string
                password:
                  type: string
                ssh_key:
                  type: string
                passphrase:
                  type: string
              required:
                - username
            directory:
              type: string
          required:
            - path
            - credentials
    DeliverConfigAzure:
      allOf:
        - $ref: '#/components/schemas/DeliverConfigBase'
        - type: object
          properties:
            type:
              enum:
                - azure
            container:
              type: string
              minLength: 3
              pattern: ^[a-z0-9](-?[a-z0-9])*$
            credentials:
              type: object
              additionalProperties: false
              properties:
                account:
                  type: string
                  pattern: ^[a-zA-Z0-9]+$
                key:
                  type: string
                  format: byte
                sas_token:
                  type: string
              required:
                - account
              oneOf:
                - required:
                    - key
                  title: Acess key
                - required:
                    - sas_token
                  title: Shared access token
            directory:
              type: string
          required:
            - container
            - credentials
    DeliverConfigBase:
      type: object
      additionalProperties: false
      properties:
        type:
          $ref: '#/components/schemas/DatasetDeliveryType'
        filename:
          type: object
          additionalProperties: false
          properties:
            template:
              type: string
              description: Template for the filename, including placeholders.
            extension:
              $ref: '#/components/schemas/DeliveredFileExt'
          required:
            - template
            - extension
      required:
        - type
        - filename
    DatasetDeliveryType:
      type: string
      description: Type of the delivery target
      enum:
        - azure
        - build
        - email
        - gcs
        - gcs_pubsub
        - s3
        - sftp
        - snowflake
        - webhook
        - ali_oss
    DeliveredFileExt:
      type: string
      enum:
        - json
        - jsonl
        - csv
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer

````