Bean Reference ============== .. note:: This reference is a work in progress and does not yet cover all available beans. For a more complete list of Heritrix beans please refer to the `javadoc `_. Core Beans ---------- ActionDirectory ~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.framework.ActionDirectory BdbCookieStore ~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.BdbCookieStore BdbFrontier ~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.frontier.BdbFrontier BdbModule ~~~~~~~~~ .. bean-doc:: org.archive.bdb.BdbModule BdbServerCache ~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.net.BdbServerCache BdbUriUniqFilter ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.util.BdbUriUniqFilter CheckpointService ~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.framework.CheckpointService CrawlController ~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.framework.CrawlController CrawlerLoggerModule ~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.reporting.CrawlerLoggerModule CrawlLimitEnforcer ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.framework.CrawlLimitEnforcer CrawlMetadata ~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.CrawlMetadata CredentialStore ~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.credential.CredentialStore DecideRuleSequence ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.DecideRuleSequence DiskSpaceMonitor ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.monitor.DiskSpaceMonitor RulesCanonicalizationPolicy ~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.canonicalize.RulesCanonicalizationPolicy SheetOverlaysManager ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.spring.SheetOverlaysManager StatisticsTracker ~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.reporting.StatisticsTracker TextSeedModule ~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.seeds.TextSeedModule Decide Rules ------------ AcceptDecideRule ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.AcceptDecideRule ClassKeyMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.deciderules.ClassKeyMatchesRegexDecideRule ContentLengthDecideRule ~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ContentLengthDecideRule ContentTypeMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ContentTypeMatchesRegexDecideRule ContentTypeNotMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ContentTypeNotMatchesRegexDecideRule ExpressionDecideRule (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ExpressionDecideRule ExternalGeoLocationDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ExternalGeoLocationDecideRule FetchStatusDecideRule ~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.FetchStatusDecideRule FetchStatusMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.FetchStatusMatchesRegexDecideRule FetchStatusNotMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.FetchStatusNotMatchesRegexDecideRule HasViaDecideRule ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.HasViaDecideRule HopCrossesAssignmentLevelDomainDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.HopCrossesAssignmentLevelDomainDecideRule HopsPathMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.HopsPathMatchesRegexDecideRule IdenticalDigestDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.recrawl.IdenticalDigestDecideRule IpAddressSetDecideRule ~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.IpAddressSetDecideRule MatchesFilePatternDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.MatchesFilePatternDecideRule MatchesListRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.MatchesListRegexDecideRule MatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.MatchesRegexDecideRule MatchesStatusCodeDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.MatchesStatusCodeDecideRule NotMatchesFilePatternDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.NotMatchesFilePatternDecideRule NotMatchesListRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.NotMatchesListRegexDecideRule NotMatchesRegexDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.NotMatchesRegexDecideRule NotMatchesStatusCodeDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.NotMatchesStatusCodeDecideRule NotOnDomainsDecideRule ~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.surt.NotOnDomainsDecideRule NotOnHostsDecideRule ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.surt.NotOnHostsDecideRule NotSurtPrefixedDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.surt.NotSurtPrefixedDecideRule OnDomainsDecideRule ~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.surt.OnDomainsDecideRule OnHostsDecideRule ~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.surt.OnHostsDecideRule PathologicalPathDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.PathologicalPathDecideRule PredicatedDecideRule ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.PredicatedDecideRule PrerequisiteAcceptDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.PrerequisiteAcceptDecideRule RejectDecideRule ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.RejectDecideRule ResourceLongerThanDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ResourceLongerThanDecideRule ResourceNoLongerThanDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ResourceNoLongerThanDecideRule ResponseContentLengthDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ResponseContentLengthDecideRule SchemeNotInSetDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.SchemeNotInSetDecideRule ScriptedDecideRule ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ScriptedDecideRule SeedAcceptDecideRule ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.SeedAcceptDecideRule SourceSeedDecideRule ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.SourceSeedDecideRule SurtPrefixedDecideRule ~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.surt.SurtPrefixedDecideRule TooManyHopsDecideRule ~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.TooManyHopsDecideRule TooManyPathSegmentsDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.TooManyPathSegmentsDecideRule TransclusionDecideRule ~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.TransclusionDecideRule ViaSurtPrefixedDecideRule ~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.deciderules.ViaSurtPrefixedDecideRule Candidate Processors -------------------- CandidateScoper ~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.prefetch.CandidateScoper FrontierPreparer ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.prefetch.FrontierPreparer Pre-Fetch Processors -------------------- PreconditionEnforcer ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.prefetch.PreconditionEnforcer Preselector ~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.prefetch.Preselector Fetch Processors ---------------- FetchDNS ~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.FetchDNS FetchFTP ~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.FetchFTP FetchHTTP ~~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.FetchHTTP FetchHTTP2 ~~~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.FetchHTTP2 FetchSFTP ~~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.FetchSFTP FetchWhois ~~~~~~~~~~ .. bean-doc:: org.archive.modules.fetcher.FetchWhois Link Extractors --------------- ExtractorCSS ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorCSS ExtractorDOC ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorDOC ExtractorHTML ~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorHTML AggressiveExtractorHTML ~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.AggressiveExtractorHTML JerichoExtractorHTML ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.JerichoExtractorHTML ExtractorHTMLForms ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.forms.ExtractorHTMLForms ExtractorHTTP ~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorHTTP ExtractorImpliedURI ~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorImpliedURI ExtractorJS ~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorJS ExtractorJson ~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorJson KnowledgableExtractorJS (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.KnowledgableExtractorJS ExtractorMultipleRegex ~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorMultipleRegex ExtractorPDF ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorPDF ExtractorPDFContent (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorPDFContent ExtractorRobotsTxt ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorRobotsTxt ExtractorSitemap ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorSitemap ExtractorSWF ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorSWF ExtractorUniversal ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorUniversal ExtractorURI ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorURI ExtractorXML ~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorXML ExtractorYoutubeDL (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeDL ExtractorYoutubeFormatStream (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeFormatStream ExtractorYoutubeChannelFormatStream (contrib) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.ExtractorYoutubeChannelFormatStream TrapSuppressExtractor ~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.extractor.TrapSuppressExtractor Browser Processor ----------------- BrowserProcessor ~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.processor.BrowserProcessor ExtractLinksBehavior ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.behaviors.ExtractLinksBehavior ScrollDownBehavior ~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.behaviors.ScrollDownBehavior Post-Processors --------------- CandidatesProcessor ~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.postprocessor.CandidatesProcessor DispositionProcessor ~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.postprocessor.DispositionProcessor ReschedulingProcessor ~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.crawler.postprocessor.ReschedulingProcessor WARCWriterChainProcessor ~~~~~~~~~~~~~~~~~~~~~~~~ .. bean-doc:: org.archive.modules.writer.WARCWriterChainProcessor DnsResponseRecordBuilder ^^^^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.DnsResponseRecordBuilder FtpControlConversationRecordBuilder ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.FtpControlConversationRecordBuilder FtpResponseRecordBuilder ^^^^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.FtpResponseRecordBuilder HttpRequestRecordBuilder ^^^^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.HttpRequestRecordBuilder HttpResponseRecordBuilder ^^^^^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.HttpResponseRecordBuilder MetadataRecordBuilder ^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.MetadataRecordBuilder RevisitRecordBuilder ^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.RevisitRecordBuilder WhoisResponseRecordBuilder ^^^^^^^^^^^^^^^^^^^^^^^^^^ .. bean-doc:: org.archive.modules.warc.WhoisResponseRecordBuilder