> ## Documentation Index
> Fetch the complete documentation index at: https://deepline.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Context.dev: Contextdev Get Web Scrape Sitemap

> List every page URL on a website from its sitemap, optionally filtered by a pattern. Includes inputs, outputs, SDK V2 examples, and Deepline cost.

List every page URL on a website from its sitemap, optionally filtered by a pattern.

> [!NOTE]
>
> Tool ID: `contextdev_get_web_scrape_sitemap`

## Run this action

Use the TypeScript SDK for one call. Use `ctx.tools.execute(...)` inside a Play for durable calls, scheduled calls, or CSV processing.

```ts theme={"languages":{"custom":["/languages/deepline-prompt.json"]}}
import { Deepline } from 'deepline';

const deepline = await Deepline.connect();
const result = await deepline.tools.execute(
  'contextdev_get_web_scrape_sitemap',
  {
    "domain": "example.com"
  },
);

console.log(result.toolResponse.raw);
```

### CLI

```bash theme={"languages":{"custom":["/languages/deepline-prompt.json"]}}
deepline tools execute contextdev_get_web_scrape_sitemap --input '{
  "domain": "example.com"
}' --json
```

## Example response

The SDK returns this shape at `result.toolResponse.raw`. The example values illustrate the response.

```json theme={"languages":{"custom":["/languages/deepline-prompt.json"]}}
{
  "data": {
    "success": true,
    "domain": "example.com",
    "urls": [
      "https://example.com"
    ],
    "meta": {
      "sitemapsDiscovered": 123,
      "sitemapsFetched": 123,
      "sitemapsSkipped": 123,
      "errors": 123
    }
  }
}
```

Use `deepline describe contextdev_get_web_scrape_sitemap --json` for the latest machine-readable contract.

## Input reference

Crawl an entire website's sitemap and return all discovered page URLs.

| Name | Type | Required | Default | Details |
| - | - | - | - | - |
| `payload.domain` | `string` | Yes | — | Domain to build a sitemap for Minimum length: 3. |
| `payload.maxLinks` | `integer` | No | `10000` | Maximum number of links to return from the sitemap crawl. Defaults to 10,000. Minimum is 1, maximum is 100,000. Minimum: 1. Maximum: 100000. |
| `payload.sitemapUrl` | `string` | No | — | Optional explicit sitemap URL. When provided, exactly this sitemap is crawled instead of discovering the domain's sitemaps. Format: `uri`. |
| `payload.urlRegex` | `string` | No | — | Optional RE2-compatible regex pattern. Only URLs matching this pattern are returned and counted against maxLinks. Maximum length: 256. |
| `payload.search` | `string` | No | — | Optional search phrase. Minimum length: 2. Maximum length: 200. |
| `payload.headers` | `record` | No | — | Optional outbound HTTP headers forwarded only to the target URL, sent as deep-object query params such as headers\[X-Custom]=value. When provided, caching is bypassed: the result is neither read from nor written to cache. |
| `payload.timeoutMS` | `integer` | No | — | Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes). Minimum: 1. Maximum: 300000. |
| `payload.zdr` | `"enabled" \| "disabled"` | No | `"disabled"` | Set to enabled to bypass shared caches and omit request and response content from retained usage logs. Requires zero data retention to be enabled for your organization (contact [support@context.dev](mailto:support@context.dev)), otherwise the request fails with ZDR\_NOT\_ENABLED. Successful ZDR responses include X-Context-ZDR: true. Allowed: `enabled`, `disabled`. |
| `payload.tags` | `array` | No | — | Optional comma-separated caller-defined tags for tracking this request. Tags are recorded on the request's usage log and can be used to filter usage on the dashboard usage page. Up to 20 tags, each 1-50 characters. Maximum items: 20. |

<details>
  <summary>Show raw input schema</summary>

  ### Input JSON Schema

  ```json theme={"languages":{"custom":["/languages/deepline-prompt.json"]}}
  {
    "type": "object",
    "description": "Crawl an entire website's sitemap and return all discovered page URLs.",
    "properties": {
      "domain": {
        "type": "string",
        "description": "Domain to build a sitemap for",
        "minLength": 3
      },
      "maxLinks": {
        "type": "integer",
        "description": "Maximum number of links to return from the sitemap crawl. Defaults to 10,000. Minimum is 1, maximum is 100,000.",
        "default": 10000,
        "minimum": 1,
        "maximum": 100000
      },
      "sitemapUrl": {
        "type": "string",
        "description": "Optional explicit sitemap URL. When provided, exactly this sitemap is crawled instead of discovering the domain's sitemaps.",
        "format": "uri"
      },
      "urlRegex": {
        "type": "string",
        "description": "Optional RE2-compatible regex pattern. Only URLs matching this pattern are returned and counted against maxLinks.",
        "maxLength": 256
      },
      "search": {
        "type": "string",
        "description": "Optional search phrase.",
        "minLength": 2,
        "maxLength": 200
      },
      "headers": {
        "type": "object",
        "description": "Optional outbound HTTP headers forwarded only to the target URL, sent as deep-object query params such as headers[X-Custom]=value. When provided, caching is bypassed: the result is neither read from nor written to cache.",
        "additionalProperties": {
          "type": "string",
          "maxLength": 8192,
          "pattern": "^[^\\r\\n]*$"
        }
      },
      "timeoutMS": {
        "type": "integer",
        "description": "Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes).",
        "minimum": 1,
        "maximum": 300000
      },
      "zdr": {
        "type": "string",
        "description": "Set to enabled to bypass shared caches and omit request and response content from retained usage logs. Requires zero data retention to be enabled for your organization (contact support@context.dev), otherwise the request fails with ZDR_NOT_ENABLED. Successful ZDR responses include X-Context-ZDR: true.",
        "default": "disabled",
        "enum": [
          "enabled",
          "disabled"
        ]
      },
      "tags": {
        "type": "array",
        "description": "Optional comma-separated caller-defined tags for tracking this request. Tags are recorded on the request's usage log and can be used to filter usage on the dashboard usage page. Up to 20 tags, each 1-50 characters.",
        "maxItems": 20,
        "items": {
          "type": "string",
          "minLength": 1,
          "maxLength": 50
        }
      }
    },
    "required": [
      "domain"
    ],
    "additionalProperties": false
  }
  ```
</details>

## Output reference

| Name | Type | Required | Default | Details |
| - | - | - | - | - |
| `result.data` | `object` | Yes | — | — |
| `result.data.success` | `true` | Yes | — | Allowed: `true`. |
| `result.data.domain` | `string` | Yes | — | — |
| `result.data.urls` | `array` | Yes | — | — |
| `result.data.meta` | `object` | Yes | — | — |
| `result.data.meta.sitemapsDiscovered` | `integer` | Yes | — | — |
| `result.data.meta.sitemapsFetched` | `integer` | Yes | — | — |
| `result.data.meta.sitemapsSkipped` | `integer` | Yes | — | — |
| `result.data.meta.errors` | `integer` | Yes | — | — |
| `result.data.key_metadata` | `object` | No | — | — |
| `result.meta` | `object` | No | — | — |

<details>
  <summary>Show raw output schema</summary>

  ### Output JSON Schema

  ```json theme={"languages":{"custom":["/languages/deepline-prompt.json"]}}
  {
    "type": "object",
    "properties": {
      "data": {
        "type": "object",
        "properties": {
          "success": {
            "type": "boolean",
            "enum": [
              true
            ]
          },
          "domain": {
            "type": "string"
          },
          "urls": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "meta": {
            "type": "object",
            "properties": {
              "sitemapsDiscovered": {
                "type": "integer"
              },
              "sitemapsFetched": {
                "type": "integer"
              },
              "sitemapsSkipped": {
                "type": "integer"
              },
              "errors": {
                "type": "integer"
              }
            },
            "required": [
              "sitemapsDiscovered",
              "sitemapsFetched",
              "sitemapsSkipped",
              "errors"
            ],
            "additionalProperties": false
          },
          "key_metadata": {
            "type": "object",
            "properties": {},
            "required": [],
            "additionalProperties": false
          }
        },
        "required": [
          "success",
          "domain",
          "urls",
          "meta"
        ],
        "additionalProperties": false
      },
      "meta": {
        "type": "object",
        "additionalProperties": true
      }
    },
    "required": [
      "data"
    ],
    "additionalProperties": false
  }
  ```
</details>

## Deepline cost

* Pricing model: `provider_usage` (provider usage).
* Estimated Deepline credits: `0.01` per pricing unit.

## Related documentation

* [Context.dev provider guide](https://deepline.com/docs/providers/contextdev/guide.md)
* [SDK V2 quickstart](https://deepline.com/docs/sdk-v2/quickstart.md)
* [SDK reference](https://deepline.com/docs/sdk-v2/sdk-reference.md)
* [Run tools across a CSV](https://deepline.com/docs/sdk-v2/batch-csv.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.