Getting Started with Heritrix¶
Heritrix is primarily used on Linux. It may run on other platforms but is not regularly tested or supported on them.
Heritrix requires Java 8 or 11. We recommend using your Linux distribution’s OpenJDK 11 packages. Alternatively up to date builds of OpenJDK 8 and 11 for several platforms are available from Adoptium.
The default Java heap for Heritrix is 256MB RAM, which is usually suitable for crawls that range over hundreds of hosts. Assign more of your available RAM to the heap if you are crawling thousands of hosts or experience Java out-of-memory problems. You can use the JAVA_OPTS variable to configure memory
Download the latest Heritrix distribution package linked from the Heritrix releases page and unzip it somewhere.
The installation will contain the following subdirectories:
- contains shell scripts/batch files for launching Heritrix.
- contains the third-party .jar files the Heritrix application requires to run.
- contains various configuration files (such as the configuration for Java logging, and pristine versions of the bundled profiles)
- the default location where operator-created jobs are stored
JAVA_HOMEenvironment variable. The value should point to your Java installation.
HERITRIX_HOMEenvironment variable. The value should be set to the path where Heritrix is installed.
Set execute permission on the Heritirix startup file.
chmod u+x $HERITRIX_HOME/bin/heritrix
To change the amount of memory allocated to Heritrix (the Java heap size), set the
JAVA_OPTSenvironment variable. The following example allocates 1GB of memory to Heritrix.
To launch Heritrix with the Web UI enabled, enter the following command. The username and password for the Web UI are set to “admin” and “admin”, respectively.
$HERITRIX_HOME/bin/heritrix -a admin:admin
By default, the Web UI listening address is only bound to the ‘localhost’ address. Therefore, the Web UI can only be accessed on the same machine from which it was launched. The ‘-b’ option may be used to listen on different/additional addresses. See Security Considerations before changing this default.
If the parameter supplied to the
-a option is a file path beginning with “@”, the admin username and password
will be read from a file. This adds an additional layer of protection to the admin username and password by ensuring
they don’t appear directly in the command-line and can’t be seen by other users running the
Accessing the User Interface¶
After Heritrix has been launched, the Web-based user interface (WUI) becomes accessible.
The URI to access the Web UI is typically
The initial login page prompts for the username and password. After login, your session will time-out after a period of non-use.
Access to the WUI is through HTTPS. Heritrix is installed with a keystore containing a self-signed certificate. This will cause your browser to display a prompt, warning that a self-signed certificate is being used. Follow the steps below for your browser to login to Heritrix for the first time.
Chrome: The message “Your connection is not private” is displayed. Click the “Advanced” button and then click “Proceed to localhost (unsafe).”
Firefox: The message “Warning: Potential Security Risk Ahead” is displayed. Click the “Advanced…” button and then click “Accept the Risk and Continue.”
Your First Crawl¶
Enter the name of the new job in the text box with the “create new job” label. Then click “create”.
Click on the name of the new job and you will be taken to the job page.
Click on the “Configuration” link at the top and the contents of the job configuration file will be displayed.
At this point you must enter several properties to make the job runnable.
- First, add the URL of page explaining how webmasters can contact you to the metadata.operatorContactUrl property.
- Next, populate the
<prop>element of the
longerOverridesbean with the seed values for the crawl. A test seed is configured for reference.
- When done click “save changes” at the top of the page.
For more detailed information on configuring jobs see Configuring Jobs and Profiles
From the job screen, click “build.” This command will validate the job configuration and load it into memory. In the Job Log the following message will display: “INFO JOB instantiated.”
Next, click the “launch” button. This command launches the job in “paused” mode. At this point the job is ready to run.
To run the job, click the “unpause” button. The job will now begin sending requests to the seeds of your crawl. The status of the job will be set to “Running.” Refresh the page to see updated statistics.
When you want to stop your crawl, click the “terminate” button to finish and then “teardown” to unload the job configuration from memory.
To exit Heritrix get back to the main page by clicking “Engine” in the top bar. Then check the “I’m sure” box under “Exit Java” and click the “exit java process” button.