> ## Documentation Index
> Fetch the complete documentation index at: https://newscatcherinc-docs.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# CatchAll API changelog
> Track updates, improvements, and fixes to the CatchAll API.
## External entity IDs, CSV exports, and connected dataset details
Adds support for your own external entity identifiers, CSV downloads for job and
monitor results, and richer connected-dataset information in pull responses.
### External entity IDs
You can now attach your own identifier to an entity with the new
`external_entity_id` field — store a CRM, data warehouse, or internal database ID
and round-trip it through CatchAll so you can join results back to your systems
without extra mapping work. The field is a free-form string (up to 255
characters) and is not enforced as unique.
**Where it appears:**
* Accepted when creating an entity via
[`POST /catchAll/entities`](/web-search-api/api-reference/entities/create-entity)
and when updating one via
[`PATCH /catchAll/entities/{entity_id}`](/web-search-api/api-reference/entities/update-entity).
* Returned on entity responses, dataset entity summaries, and connected entities
in Company Watchlist results. Null when not set.
### CSV export endpoints
Pull job and monitor results directly as a CSV download for spreadsheet-friendly
analysis and reporting. Each row is one record, with enrichment fields as
columns, citations as a JSON column, and connected entities split into
`event_associated_entities` and `mention_entities` JSON columns.
**New endpoints:**
* [`GET /catchAll/pull/{job_id}/csv`](/web-search-api/api-reference/jobs/get-job-results-as-csv)
— Download a completed job's results as CSV.
* [`GET /catchAll/monitors/pull/{monitor_id}/csv`](/web-search-api/api-reference/monitors/get-latest-monitor-results-as-csv)
— Download the latest monitor run's results as CSV.
### Connected dataset details in pull responses
Job and monitor pull responses now include a `connected_datasets` array — each
entry has the dataset `id`, `name`, and an `is_deleted` flag — alongside an
`is_all_news_query` boolean indicating whether the query was submitted as an
all-news (watchlist-generic) query.
### Fixes and improvements
* Reliability and observability improvements, data-model and validation
hardening, and language-query guardrails.
* Relaxed company domain requirements and safer enrichment handling for edge
cases.
## Company Watchlist improvements and performance fixes
Adds two new Company Watchlist controls — association-type filtering and
all-news queries — along with performance improvements and API quality fixes.
### Association type filtering
When submitting a job connected to a dataset (watchlist), you can now set
`ed_association_type` to control how strictly tracked entities must relate to
each returned event.
* **`event_associated`** — only returns events where a tracked entity is the
primary actor (e.g., the company that raised funding, was acquired, or
announced layoffs). Passing references are excluded.
* **`mention`** — returns any event where a tracked entity appears, whether as
a primary actor or a passing reference.
When omitted, no association-type filter is applied.
Each matched entity in results includes an `association_type` field indicating
how it relates to that specific record.
**New parameter:**
* `ed_association_type` on
[`POST /catchAll/submit`](/web-search-api/api-reference/jobs/create-job)
— Filter connected-entity results by association type. Requires
`connected_dataset_ids`.
### All-news watchlist queries
You can now submit a connected job without specifying a topic query. Set
`fetch_all_watchlist_news: true` to retrieve all news for the entities in your
connected watchlist — no topic required. Wildcard queries (`*`) and generic
phrases like "all news" also trigger this behavior.
Monitors created from an all-news job preserve the behavior: each recurring run
collects all news for the connected watchlist, not topic-restricted results.
`fetch_all_watchlist_news` without `connected_dataset_ids` returns a validation
error.
**New parameters:**
* `fetch_all_watchlist_news` on
[`POST /catchAll/submit`](/web-search-api/api-reference/jobs/create-job)
— When `true`, retrieves all news for connected watchlist entities without
topic filtering. Requires `connected_dataset_ids`.
* `fetch_all_watchlist_news` on
[`POST /catchAll/initialize`](/web-search-api/api-reference/jobs/initialize-job)
— Preview validators and enrichments for an all-news watchlist query. Also
accepts `connected_dataset_ids`.
### Fixes and improvements
* **Performance**: Faster percolation queries, reduced Elasticsearch pressure,
and lower HTTP client concurrency for more stable throughput under load.
* **API correctness**: Schema type cleanup and enum-based job-status retrieval.
Improved OpenAPI spec to enable automated SDK generation.
* **Project deletion**: Better notification handling and improved response
structure when deleting a project with resources.
## Centralized webhooks, query validation, and projects
Introduces centralized webhook management to replace per-monitor inline
configuration, a pre-submission query validation endpoint, and projects for
organizing and sharing resources.
### Centralized webhooks
Replaces the `webhook` inline field on monitor creation and update with a
reusable, resource-agnostic webhook system.
**Features:**
* **Webhook management**: Create named webhook endpoints at the organization
level and reuse them across any number of jobs or monitors.
* **Flexible assignment**: Pass `webhook_ids` at job or monitor creation time,
or assign webhooks after creation. Each resource supports up to 5 webhooks.
* **Delivery modes**: `full` sends all records in a single request per trigger.
`per_record` sends one request per record.
* **Authentication**: Bearer token, API key header, and basic auth forwarded
with each delivery request.
* **Platform-formatted payloads**: `slack` and `teams` types send pre-formatted
Slack Block Kit and Microsoft Teams Adaptive Card payloads automatically.
* **Test delivery**: Verify endpoint reachability and authentication before
attaching a webhook to a live resource.
* **Delivery history**: Inspect per-dispatch outcomes, HTTP status codes, and
error messages for any job or monitor.
**New endpoints:**
* [`POST /catchAll/webhooks`](/web-search-api/api-reference/webhooks/create-webhook)
— Create a webhook.
* [`GET /catchAll/webhooks`](/web-search-api/api-reference/webhooks/list-webhooks)
— List webhooks.
* [`GET /catchAll/webhooks/{webhook_id}`](/web-search-api/api-reference/webhooks/get-webhook)
— Get a webhook.
* [`PATCH /catchAll/webhooks/{webhook_id}`](/web-search-api/api-reference/webhooks/update-webhook)
— Update a webhook.
* [`DELETE /catchAll/webhooks/{webhook_id}`](/web-search-api/api-reference/webhooks/delete-webhook)
— Delete a webhook.
* [`POST /catchAll/webhooks/{webhook_id}/test`](/web-search-api/api-reference/webhooks/test-webhook)
— Test webhook delivery.
* [`POST /catchAll/webhooks/{webhook_id}/resources`](/web-search-api/api-reference/webhooks/assign-resource)
— Assign a resource to a webhook.
* [`GET /catchAll/webhooks/{webhook_id}/resources`](/web-search-api/api-reference/webhooks/list-webhook-resources)
— List resources assigned to a webhook.
* [`DELETE /catchAll/webhooks/{webhook_id}/resources/{resource_type}/{resource_id}`](/web-search-api/api-reference/webhooks/remove-resource)
— Remove a resource from a webhook.
* [`GET /catchAll/resources/{resource_type}/{resource_id}/webhooks`](/web-search-api/api-reference/webhooks/list-resource-webhooks)
— List webhooks assigned to a resource.
* [`GET /catchAll/webhook-history`](/web-search-api/api-reference/webhooks/get-delivery-history)
— Get webhook delivery history for a resource.
### Query validation
Adds a pre-submission endpoint that checks whether a query is well-formed
before creating a job.
**New endpoint:**
* [`POST /catchAll/validate`](/web-search-api/api-reference/jobs/validate-query)
— Validate a query before submission.
### Projects
Organizes jobs, monitors, and datasets into named containers that can be
shared with teammates.
**Features:**
* **Resource grouping**: Assign resources to a project at creation time via
`project_id`, or post-hoc using the project resources endpoints.
* **Project filtering**: Filter jobs, monitors, and datasets by project using
the `project_id` query parameter on all list endpoints.
* **Project overview**: Get a resource status breakdown without fetching the
full resource list.
**Endpoints:**
* [`GET /catchAll/projects`](/web-search-api/api-reference/projects/list-projects)
— List projects.
* [`POST /catchAll/projects`](/web-search-api/api-reference/projects/create-project)
— Create a project.
* [`GET /catchAll/projects/{project_id}`](/web-search-api/api-reference/projects/get-project)
— Get a project.
* [`PATCH /catchAll/projects/{project_id}`](/web-search-api/api-reference/projects/update-project)
— Update a project.
* [`DELETE /catchAll/projects/{project_id}`](/web-search-api/api-reference/projects/delete-project)
— Delete a project.
* [`GET /catchAll/projects/{project_id}/overview`](/web-search-api/api-reference/projects/get-project-overview)
— Get resource counts by type and status.
* [`GET /catchAll/projects/{project_id}/resources`](/web-search-api/api-reference/projects/list-resources)
— List resources in a project.
* [`POST /catchAll/projects/{project_id}/resources`](/web-search-api/api-reference/projects/add-resource)
— Add resources to a project.
* [`DELETE /catchAll/projects/{project_id}/resources/{resource_type}/{resource_id}`](/web-search-api/api-reference/projects/remove-resource)
— Remove a resource from a project.
## ED score filtering, entity attributes, and monitor timezone
Refines Company Watchlist with relevance threshold control and richer entity
data in results; adds timezone support for monitor scheduling.
**Features:**
* **ED score filtering**: Submit jobs with `ed_score_min` to exclude connected
entities below a relevance threshold. Only records where at least one entity
meets the minimum score are returned.
* **Entity attributes in results**: Each matched entity in `connected_entities`
now includes a `type` field and a `company` object with domain, alternative
names, and key persons.
* **Monitor timezone**: Set an explicit IANA timezone on monitor creation with
the new `timezone` parameter. Used as the fallback when the schedule string
does not include one. Defaults to `"UTC"`.
**Changed:**
* [`POST /catchAll/datasets/{dataset_id}/entities/list`](/web-search-api/api-reference/datasets/list-entities-in-dataset)
replaces `GET /catchAll/datasets/{dataset_id}/entities` for listing entities.
Pagination and filter parameters are now accepted in the request body.
* [`POST /catchAll/datasets/upload`](/web-search-api/api-reference/datasets/create-dataset-from-csv)
and
[`POST /catchAll/datasets/{dataset_id}/upload`](/web-search-api/api-reference/datasets/upload-csv-to-dataset)
now require a `domain` column. A missing column header returns `422`. Rows
where `domain` is empty are skipped and reported in `validation_report`.
## Resource management and sharing
Introduces job and monitor deletion, monitor execution history, resource sharing,
and filtering by ownership across all list endpoints.
**Features:**
* **Delete jobs and monitors**: Remove jobs and monitors you no longer need.
Deleted resources no longer appear in list results. Deletion is idempotent —
deleting an already-deleted resource returns `200`.
* **Monitor status history**: Inspect the full execution history of a monitor,
including per-run record counts and webhook delivery results.
* **Resource sharing**: Jobs, monitors, and datasets shared with you by
organization members now appear in list responses with a `sharing_info` object
indicating who shared the resource, when, and with what permission level.
* **Ownership filter**: All list endpoints for jobs, monitors, and datasets
support a new `ownership` parameter (`all`, `own`, `shared`) to filter results
by who owns or shared them. Defaults to `all` for backward compatibility.
* **Search filter**: Filter jobs and monitors by text using the new `search`
parameter on list endpoints.
**Endpoints:**
* [`DELETE /catchAll/jobs/{job_id}`](/web-search-api/api-reference/jobs/delete-job)
— Delete a job.
* [`DELETE /catchAll/monitors/{monitor_id}`](/web-search-api/api-reference/monitors/delete-monitor)
— Delete a monitor.
* [`GET /catchAll/monitors/{monitor_id}/status`](/web-search-api/api-reference/monitors/get-monitor-status-history)
— Get monitor execution status history.
## Company search
Introduces company search — track specific companies across CatchAll jobs by
connecting datasets of entities.
**Features:**
* **Entities**: Define companies with names, domains, alternative names, and key
persons.
* **Datasets**: Group entities into reusable collections (watchlists,
portfolios).
* **Connected jobs**: Submit jobs with `connected_dataset_ids` to activate
company search mode.
* **Per-company scoring**: Each result record includes a `connected_entities`
array with `entity_id`, `name`, `ed_score`, and `relation` for each matched
company.
**Endpoints:**
* [`GET /catchAll/entities`](/web-search-api/api-reference/entities/list-entities)
— List entities.
* [`POST /catchAll/entities`](/web-search-api/api-reference/entities/create-entity)
— Create a company entity.
* [`POST /catchAll/entities/batch`](/web-search-api/api-reference/entities/create-entities-batch)
— Create multiple entities.
* [`GET /catchAll/entities/{entity_id}`](/web-search-api/api-reference/entities/get-entity)
— Get entity.
* [`PATCH /catchAll/entities/{entity_id}`](/web-search-api/api-reference/entities/update-entity)
— Update entity.
* [`DELETE /catchAll/entities/{entity_id}`](/web-search-api/api-reference/entities/delete-entity)
— Delete entity.
* [`GET /catchAll/datasets`](/web-search-api/api-reference/datasets/list-datasets)
— List datasets.
* [`POST /catchAll/datasets`](/web-search-api/api-reference/datasets/create-dataset)
— Create dataset from entity IDs.
* [`POST /catchAll/datasets/upload`](/web-search-api/api-reference/datasets/create-dataset-from-csv)
— Create dataset from CSV.
* [`GET /catchAll/datasets/{dataset_id}`](/web-search-api/api-reference/datasets/get-dataset)
— Get dataset.
* [`PATCH /catchAll/datasets/{dataset_id}`](/web-search-api/api-reference/datasets/update-dataset)
— Update dataset.
* [`DELETE /catchAll/datasets/{dataset_id}`](/web-search-api/api-reference/datasets/delete-dataset)
— Delete dataset.
* [`GET /catchAll/datasets/{dataset_id}/entities`](/web-search-api/api-reference/datasets/list-entities-in-dataset)
— List entities in dataset.
* [`POST /catchAll/datasets/{dataset_id}/entities`](/web-search-api/api-reference/datasets/add-entities-to-dataset)
— Add entities to dataset.
* [`DELETE /catchAll/datasets/{dataset_id}/entities`](/web-search-api/api-reference/datasets/remove-entities-from-dataset)
— Remove entities from dataset.
* [`GET /catchAll/datasets/{dataset_id}/status`](/web-search-api/api-reference/datasets/get-dataset-status-history)
— Get dataset status history.
* [`POST /catchAll/datasets/{dataset_id}/upload`](/web-search-api/api-reference/datasets/upload-csv-to-dataset)
— Add companies to dataset via CSV.
**Changed:**
* [`POST /catchAll/submit`](/web-search-api/api-reference/jobs/create-job) now
accepts `connected_dataset_ids` — an optional array of dataset UUIDs that
activates company search mode.
## Job modes, plan limits, and response improvements
**Features:**
* **Job modes**: Submit jobs in `lite` or `base` mode with the new `mode`
parameter. Lite mode provides lightweight extraction with titles and citations
only. Base mode runs the full pipeline with validation and enrichment.
* **Plan limits**: Retrieve plan features and current usage, including
concurrency slots, max results, lookback limits, and monthly credits.
**Endpoints:**
* [`POST /catchAll/user/limits`](/web-search-api/api-reference/meta/get-plan-limits)
— Get plan features and current usage.
**Improvements:**
* [`GET /catchAll/pull/{job_id}`](/web-search-api/api-reference/jobs/get-job-results)
now returns `mode` and full validator/enrichment objects instead of name-only
arrays.
* [`GET /catchAll/jobs/user`](/web-search-api/api-reference/jobs/list-user-jobs)
now returns `mode` and masked `user_key` for each job.
* [`GET /catchAll/monitors`](/web-search-api/api-reference/monitors/list-monitors)
now returns masked `user_key` for each monitor.
* Fixed duplicate confidence values appearing in enrichment records.
* Fixed pagination handling in the pull endpoint.
* Improved multi-word query handling for lite mode.
* Gateway authentication caching reduces response times.
## Monitor controls and response improvements
**Features:**
* **Monitor limit**: Control the maximum number of records per monitor run with
the new `limit` parameter.
* **Monitor backfill**: Choose whether to fill the data gap before the first
scheduled run with the new `backfill` parameter.
**Improvements:**
* [`GET /catchAll/monitors`](/web-search-api/api-reference/monitors/list-monitors)
now supports `page` and `page_size` parameters and sorts by creation date.
* [`GET /catchAll/pull/{job_id}`](/web-search-api/api-reference/jobs/get-job-results)
now returns the `error` field for failed jobs and the `limit` applied to the
job.
* [`POST /catchAll/continue`](/web-search-api/api-reference/jobs/continue-job)
now correctly enforces result limits. `new_limit` defaults to your plan
maximum if omitted.
* [`GET /catchAll/jobs/user`](/web-search-api/api-reference/jobs/list-user-jobs)
now returns a structured response object with pagination metadata instead of a
flat array.
* Consistent status representation across the status and pull endpoints.
* Improved caching and overall response performance.
* Submitting a validator or enrichment with an empty `name` now returns a
validation error.
## Stable release with enhanced control and structured company enrichments
CatchAll API reaches v1.0.0 with improved query customization, date range
controls, and structured company enrichments.
**Features:**
* **Initialize endpoint**: Get suggested validators, enrichments, and date
ranges before submitting jobs. Returns `date_modification_message` when
requested dates exceed plan limits.
* **Date range controls**: Specify `start_date` and `end_date` in submit
requests. System validates against plan limits and returns 400 errors for
invalid ranges.
* **Custom validators and enrichments**: Define custom validation criteria and
extraction fields directly in submit requests instead of relying on automatic
generation.
* **Company enrichment type**: Extract structured company data including name,
alternative names, website candidates, people, and address using the new
`company` enrichment type.
* **Paginated user jobs**: List user jobs endpoint now supports `page` and
`page_size` parameters for efficient retrieval of job history.
**Endpoints:**
* [`POST /catchAll/initialize`](/web-search-api/api-reference/jobs/initialize-job)
— Get validator, enrichment, and date range suggestions.
* Enhanced
[`POST /catchAll/submit`](/web-search-api/api-reference/jobs/create-job) — Now
accepts date ranges, validators, and enrichments.
* Enhanced
[`GET /catchAll/jobs/user`](/web-search-api/api-reference/jobs/list-user-jobs)
— Now supports pagination.
**Improvements:**
* Enhanced deduplication logic reduces duplicate records across job executions.
* Status endpoint returns more accurate current processing stage.
* User jobs endpoint returns results sorted by creation date (most recent
first).
## Job continuation and early result access
**Features:**
* **Job continuation**: Submit jobs with the optional `limit` parameter for fast
testing. Review results and continue jobs with `/catchAll/continue` to process
more records without restarting.
* **Early result access**: Pull results during the `enriching` stage without
waiting for job completion. A new `progress_validated` field shows validation
progress as results become available in batches.
* **Monitor updates**: Update webhook configuration for existing monitors
without recreating them.
**Endpoints:**
* [`POST /catchAll/continue`](/web-search-api/api-reference/jobs/continue-job) —
Continue completed job with higher limit.
* [`PATCH /catchAll/monitors/{monitor_id}`](/web-search-api/api-reference/monitors/update-monitor)
— Update monitor webhook configuration.
**Improvements:**
* Expanded search range provides broader coverage for improved recall.
## API response enhancements and monitor configuration
**Features:**
* **Enrichments metadata**: Job pull endpoint returns `enrichments` array
listing all extracted field names.
* **Human-readable schedules**: Monitor list includes `schedule_human_readable`
alongside cron expressions.
* **Webhook visibility**: Monitor list exposes full webhook configuration
including URL, method, and headers.
**Improvements:**
* Pull endpoints return all citations without limits.
* Monitors require minimum 24-hour intervals between executions.
* First webhook execution includes complete reference job results.
* Improved validation error messages for invalid schedules.
## Status tracking and monitor improvements
Enhanced status tracking with processing steps and improved monitor
functionality for time-constrained queries.
**Features:**
* **Progress tracking**: Status endpoint now returns detailed
[`steps`](/web-search-api/api-reference/jobs/get-job-status#response-steps)
array showing completion state for each processing stage.
* **Monitor timestamps**: Monitor records include `added_on` and `updated_on`
fields to track when each data record was first collected and last updated
across job executions.
* **Time-constrained jobs**: Monitors now work with reference jobs that include
time constraints. The system automatically adapts date-specific validators for
each execution.
**Improvements:**
* [`GET /catchAll/jobs/user`](/web-search-api/api-reference/jobs/list-user-jobs)
now returns only user-created jobs, excluding repeated monitor jobs.
* Monitor responses use
[`run_info`](/web-search-api/api-reference/monitors/get-monitor-results#response-run-info)
structure with `first_run` and `last_run` timestamps instead of `date_range`
for clearer execution metadata.
* Improved error handling in monitor creation endpoint with specific validation
messages for invalid schedules and incomplete reference jobs.
## Monitor webhooks and deduplication
Introduced webhook notifications and intelligent deduplication for monitors to
streamline automated data collection workflows.
**Features:**
* **Webhook notifications**: Receive real-time POST notifications when scheduled
monitor jobs complete, including monitor ID, latest job ID, and record counts.
* **Intelligent deduplication**: Monitors automatically eliminate duplicate
records across job executions, sending only new results to webhooks.
* **Monitor management**: Enable or disable monitors to control automated job
execution.
**Endpoints:**
* [`GET /catchAll/monitors`](/web-search-api/api-reference/monitors/list-monitors)
— List all monitors for your API key.
* [`POST /catchAll/monitors/{monitor_id}/enable`](/web-search-api/api-reference/monitors/enable-monitor)
— Enable a disabled monitor.
* [`POST /catchAll/monitors/{monitor_id}/disable`](/web-search-api/api-reference/monitors/disable-monitor)
— Disable an active monitor.
**Improvements:**
* Reduced default date range from 14 days to 5 days for faster job processing.
* Limited maximum date range to 30 days to maintain optimal performance.
* Fixed special characters displaying incorrectly in web page content.
* Improved error handling when monitor jobs have incomplete data.
## Pagination and reliability improvements
Enhanced core functionality with pagination fixes, job tracking capabilities,
and improved data quality.
**Endpoints:**
* [`GET /catchAll/jobs/user`](/web-search-api/api-reference/jobs/list-user-jobs)
— List all jobs created with your API key.
* [`GET /catchAll/monitors/pull/{monitor_id}`](/web-search-api/api-reference/monitors/get-monitor-results)
— Retrieve aggregated results from all monitor jobs.
**Improvements:**
* Fixed pagination returning all records instead of the requested page.
* Enhanced pagination support for handling large result sets with `page` and
`page_size` parameters.
* Fixed jobs failing to complete during data extraction.
* Fixed timezone errors when scheduling monitor jobs.
* Fixed monitors failing to execute all configured queries.
## Monitors feature
Introduced
[monitors for automated queries](/web-search-api/concepts/monitors) —
schedule recurring data collection from a reference job.
**Features:**
* **Schedule automation**: Create monitors that run queries on schedules defined
in plain text format (e.g., "every day at 12 PM UTC").
* **Job tracking**: Track all jobs associated with a monitor.
* **Execution history**: Retrieve job execution history and status for each
monitor.
* **Webhook notifications**: Get notified when scheduled jobs complete.
**Endpoints:**
* [`POST /catchAll/monitors/create`](/web-search-api/api-reference/monitors/create-monitor)
— Create scheduled monitor.
* [`GET /catchAll/monitors/{monitor_id}/jobs`](/web-search-api/api-reference/monitors/list-monitor-jobs)
— List monitor jobs.
**Improvements:**
* Added date range information to results showing the time period covered.
* Fixed incorrect date filtering in search results.
## Initial beta release
CatchAll is a web search API that transforms text queries into structured,
validated event data extracted from billions of web sources.
**Features:**
* **Plain text input**: Submit queries in plain text and receive structured
event data with source citations.
* **[Dynamic schemas](/web-search-api/concepts/dynamic-schemas)**:
Response structures adapt to your query, generating relevant fields
automatically.
* **Real-time status tracking**: Track job progress from submission through
completion, including analysis, retrieval, clustering, validation, and
extraction.
**Endpoints:**
* [`POST /catchAll/submit`](/web-search-api/api-reference/jobs/create-job) —
Create jobs using a plain text queries.
* [`GET /catchAll/status/{job_id}`](/web-search-api/api-reference/jobs/get-job-status)
— Check a job processing status.
* [`GET /catchAll/pull/{job_id}`](/web-search-api/api-reference/jobs/get-job-results)
— Retrieve structured results with citations.
→
[Get started with the quickstart guide](/web-search-api/get-started/quickstart)