DuckDuckGo Tracker Radar Collector

🕸 Modular, multithreaded, puppeteer-based crawler used to generate third party request data for the Tracker Radar.

1. Clone this project locally (git clone git@github.com:duckduckgo/tracker-radar-collector.git)
2. Install all dependencies (npm i)
3. Run the command line tool:

npm run crawl -- -u "https://example.com" -o ./data/ -v

Available options:

• -o, --output <path> - (required) output folder where output files will be created
• -u, --url <url> - single URL to crawl
• -i, --input-list <path> - path to a text file with list of URLs to crawl (each in a separate line)
• -d, --data-collectors <list> - comma separated list (e.g -d 'requests,cookies') of data collectors that should be used (all by default)
• -c, --crawlers <number> - override the default number of concurrent crawlers (default number is picked based on the number of CPU cores)
• --reporters <list> - comma separated list (e.g. --reporters 'cli,file,html') of reporters to be used ('cli' by default)
• -v, --verbose - instructs reporters to log additional information (e.g. for "cli" reporter progress bar will not be shown when verbose logging is enabled)
• -l, --log-path <path> - instructs reporters where all logs should be written to
• -f, --force-overwrite - overwrite existing output files (by default entries with existing output files are skipped)
• -3, --only-3p - don't save any first-party data (e.g. requests, API calls for the same eTLD+1 as the main document)
• -m, --mobile - emulate a mobile device when crawling
• -p, --proxy-config <host> - optional SOCKS proxy host
• -r, --region-code <region> - optional 2 letter region code. For metadata only
• -a, --disable-anti-bot - disable simple build-in anti bot detection script injected to every frame
• --chromium-version <version_number> - use custom version of Chromium (e.g. "843427") instead of using the default
• --selenium-hub <url> - If provided, browsers will be requested from selenium hub instead of spawning local processes (e.g. --selenium-hub http://my-selenium-hub-host:4444).
• --config <path> - path to a config file that allows to set all the above settings (and more). Note that CLI flags have a higher priority than settings passed via config. You can find a sample config file in tests/cli/sampleConfig.json.
• --autoconsent-action <action> - automatic autoconsent action (requires the cookiepopups collector). Possible values: optIn, optOut

1. Install this project as a dependency (npm i git+https://github.com:duckduckgo/tracker-radar-collector.git).

2. Import it:

// you can either import a "crawlerConductor" that runs multiple crawlers for you
const {crawlerConductor} = require('tracker-radar-collector');
// or a single crawler
const {crawler} = require('tracker-radar-collector');

// you will also need some data collectors (/collectors/ folder contains all build-in collectors)
const {RequestCollector, CookieCollector, …} = require('tracker-radar-collector');

3. Use it:

crawlerConductor({
    // required ↓
    urls: ['https://example.com', {url: 'https://duck.com', dataCollectors: [new ScreenshotCollector()]}, …], // two formats available: first format will use default collectors set below, second format will use custom set of collectors for this one url
    dataCallback: (url, result) => {…},
    // optional ↓
    dataCollectors: [new RequestCollector(), new CookieCollector()],
    failureCallback: (url, error) => {…},
    numberOfCrawlers: 12,// custom number of crawlers (there is a hard limit of 38 though)
    logFunction: (...msg) => {…},// custom logging function
    filterOutFirstParty: true,// don't save any first-party data (false by default)
    emulateMobile: