This guide covers best practices for monitor configuration, webhook
implementation, and performance optimization.
Choose appropriate schedules
Match schedule frequency to your data needs and budget. Each scheduled run creates a billable job.
| Use Case | Schedule | Rationale |
|---|
| Event monitoring | Daily | Balances freshness with cost |
| Regulatory updates | Daily or weekly | Changes occur infrequently |
| Market intelligence | Daily or weekly | Adjust based on update cycles |
Minimum default interval: 24 hours. Schedules more frequent depends on your subscription plan.
Define schedules in natural language with explicit timezone.
"every day at 12 PM UTC"
"every day at 9 AM EST"
"every Monday at 9 AM EST"
"every Friday at 5 PM GMT"
"every 2 days at 3 PM UTC"
"every 3 days at 9 AM EST"
"every 7 days at 12 PM UTC"
"every hour"
"every 6 hours"
"every 12 hours"
Common cron patterns:
| Schedule | Cron Expression | Valid |
|---|
"every day at 12 PM UTC" | 0 12 * * * | ✅ Daily |
"every Monday at 9 AM EST" | 0 9 * * 1 | ✅ Weekly |
"every 6 hours" | 0 */6 * * * | ✅ Hourly (For specific plans) |
Testing procedure
Test your schedule format before production use.
Create test monitor
Create a monitor with your desired schedule (e.g., "every day at 3 PM UTC")
Wait for execution
Wait for the scheduled time to pass.
Verify cron expression
Check the cron_expression field:curl "https://catchall.newscatcherapi.com/catchAll/monitors/{monitor_id}/jobs" \
-H "x-api-key: YOUR_API_KEY"
Create production monitor
If correct, create your production monitor. If incorrect, adjust the schedule format.
Implement robust webhooks
Configure webhook endpoints to handle notifications reliably.
Endpoint requirements
Your webhook endpoint must:
- Return 2xx status code within 5 seconds.
- Be publicly accessible (not localhost or private network).
- Use HTTPS (not HTTP).
- Handle POST requests with JSON body.
Quick implementation
Return 200 immediately and process asynchronously to avoid timeouts:
from flask import Flask, request, jsonify
import logging
app = Flask(__name__)
logging.basicConfig(level=logging.INFO)
@app.route('/catchall/webhook', methods=['POST'])
def handle_catchall_webhook():
try:
# Get payload
payload = request.json
logging.info(f"Received webhook: {payload['monitor_id']}")
# Return 200 immediately - process async
process_webhook_async(payload)
return jsonify({"status": "received"}), 200
except Exception as e:
logging.error(f"Webhook error: {e}")
# Return 200 even on error to avoid retries
return jsonify({"status": "error"}), 200
def process_webhook_async(payload):
"""Queue for background processing"""
monitor_id = payload['monitor_id']
records_count = payload['records_count']
if records_count > 0:
# Your processing logic here
save_records(payload['records'])
import express from "express";
const app = express();
app.use(express.json());
app.post("/catchall/webhook", async (req, res) => {
try {
// Get payload
const payload = req.body;
console.log(`Received webhook: ${payload.monitor_id}`);
// Return 200 immediately
res.status(200).json({ status: "received" });
// Process asynchronously (don't await)
processWebhookAsync(payload);
} catch (error) {
console.error("Webhook error:", error);
// Return 200 even on error to avoid retries
res.status(200).json({ status: "error" });
}
});
async function processWebhookAsync(payload: any) {
const { monitor_id, records_count } = payload;
if (records_count > 0) {
// Your processing logic here
await saveRecords(payload.records);
}
}
app.listen(3000, () => {
console.log("Webhook server running on port 3000");
});
import org.springframework.web.bind.annotation.*;
import org.springframework.http.ResponseEntity;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
@RestController
@RequestMapping("/catchall")
public class WebhookController {
private static final Logger logger = LoggerFactory.getLogger(WebhookController.class);
@PostMapping("/webhook")
public ResponseEntity<Map<String, String>> handleWebhook(@RequestBody Map<String, Object> payload) {
try {
// Get payload
String monitorId = (String) payload.get("monitor_id");
logger.info("Received webhook: {}", monitorId);
// Return 200 immediately - process async
processWebhookAsync(payload);
return ResponseEntity.ok(Map.of("status", "received"));
} catch (Exception e) {
logger.error("Webhook error: ", e);
// Return 200 even on error to avoid retries
return ResponseEntity.ok(Map.of("status", "error"));
}
}
private void processWebhookAsync(Map<String, Object> payload) {
CompletableFuture.runAsync(() -> {
String monitorId = (String) payload.get("monitor_id");
Integer recordsCount = (Integer) payload.get("records_count");
if (recordsCount != null && recordsCount > 0) {
// Your processing logic here
saveRecords((List<?>) payload.get("records"));
}
});
}
}
Show Add retry logic with exponential backoff
Implement exponential backoff for webhook processing failures:import time
def process_webhook_with_retry(payload, max_retries=3):
"""Process webhook with exponential backoff"""
for attempt in range(max_retries):
try:
# Your processing logic
process_records(payload['records'])
return True
except Exception as e:
if attempt < max_retries - 1:
wait_time = 2 ** attempt # Exponential backoff: 1s, 2s, 4s
time.sleep(wait_time)
continue
else:
# Log failure after all retries
log_webhook_failure(payload, str(e))
return False
async function processWebhookWithRetry(
payload: any,
maxRetries: number = 3
): Promise<boolean> {
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
// Your processing logic
await processRecords(payload.records);
return true;
} catch (error) {
if (attempt < maxRetries - 1) {
const waitTime = Math.pow(2, attempt) * 1000; // 1s, 2s, 4s
await new Promise((resolve) => setTimeout(resolve, waitTime));
continue;
} else {
// Log failure after all retries
await logWebhookFailure(payload, error.message);
return false;
}
}
}
return false;
}
import java.util.Map;
import java.util.List;
public boolean processWebhookWithRetry(Map<String, Object> payload, int maxRetries) {
for (int attempt = 0; attempt < maxRetries; attempt++) {
try {
// Your processing logic
processRecords((List<?>) payload.get("records"));
return true;
} catch (Exception e) {
if (attempt < maxRetries - 1) {
long waitTime = (long) Math.pow(2, attempt) * 1000; // 1s, 2s, 4s
try {
Thread.sleep(waitTime);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
return false;
}
} else {
// Log failure after all retries
logWebhookFailure(payload, e.getMessage());
return false;
}
}
}
return false;
}
Log all webhook events for debugging:import json
from datetime import datetime
def log_webhook(payload, status):
"""Log webhook receipt and processing status"""
log_entry = {
"timestamp": datetime.utcnow().isoformat(),
"monitor_id": payload['monitor_id'],
"latest_job_id": payload['latest_job_id'],
"records_count": payload['records_count'],
"status": status
}
with open('webhook_log.jsonl', 'a') as f:
f.write(json.dumps(log_entry) + '\n')
import * as fs from "fs";
function logWebhook(payload: any, status: string): void {
const logEntry = {
timestamp: new Date().toISOString(),
monitor_id: payload.monitor_id,
latest_job_id: payload.latest_job_id,
records_count: payload.records_count,
status: status,
};
fs.appendFileSync(
"webhook_log.jsonl",
JSON.stringify(logEntry) + "\n",
"utf8"
);
}
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.FileWriter;
import java.io.IOException;
import java.time.Instant;
import java.util.HashMap;
import java.util.Map;
public void logWebhook(Map<String, Object> payload, String status) {
Map<String, Object> logEntry = new HashMap<>();
logEntry.put("timestamp", Instant.now().toString());
logEntry.put("monitor_id", payload.get("monitor_id"));
logEntry.put("latest_job_id", payload.get("latest_job_id"));
logEntry.put("records_count", payload.get("records_count"));
logEntry.put("status", status);
try (FileWriter writer = new FileWriter("webhook_log.jsonl", true)) {
ObjectMapper mapper = new ObjectMapper();
writer.write(mapper.writeValueAsString(logEntry) + "\n");
} catch (IOException e) {
logger.error("Failed to write webhook log", e);
}
}
Update monitor configuration
Update webhook assignments or the record limit for an existing monitor without
recreating it.
When to update
Update a monitor when you need to:
- Change which webhooks receive notifications
- Clear all webhook assignments
- Adjust the maximum records per run
Schedule and reference job cannot be modified. To change the query or schedule,
create a new monitor.
Update procedure
Get the webhook ID
If you don’t have the webhook ID, list your webhooks: curl "https://catchall.newscatcherapi.com/catchAll/webhooks" \
-H "x-api-key: YOUR_API_KEY"
Send update request
curl -X PATCH "https://catchall.newscatcherapi.com/catchAll/monitors/{monitor_id}" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"webhook_ids": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890"],
"limit": 100
}'
To clear all webhook assignments, pass an empty array: "webhook_ids": [].Verify update
List your monitors to confirm the change took effect: curl "https://catchall.newscatcherapi.com/catchAll/monitors" \
-H "x-api-key: YOUR_API_KEY"
Updates take effect immediately. The next scheduled execution uses the new
configuration.
See also