Class FetchEngine

constructor

• new FetchEngine(options?: EngineOptions): FetchEngine

  Throws OpenKaValidationError for an option out of range (assertEngineOptions).

  Parameters

  • Optionaloptions: EngineOptions

  Returns FetchEngine

ReadonlymaxResponseBytes

ReadonlyuserAgent

metrics

• get metrics(): Readonly<Omit<EngineMetrics, "durations">> & {
      durations: readonly number[];
  }

  Where this engine's time went so far (EngineMetrics), live; callers read it.

  Returns Readonly<Omit<EngineMetrics, "durations">> & { durations: readonly number[] }

get

• get(
      pathOrUrl: string,
      options?: {
          headers?: Record<string, string>;
          maxRetries?: number;
          onRedirect?: RedirectHook;
          params?: QueryParams;
          validators?: CacheValidators;
      },
  ): Promise<FetchResult>

  GET a URL, honouring rate limits, retries, redirects and conditional requests. A 304 comes back as notModified rather than an error, because "unchanged" is the common and cheap case the pipeline is built around.

  Parameters

  • pathOrUrl: string
  • Optionaloptions: {
        headers?: Record<string, string>;
        maxRetries?: number;
        onRedirect?: RedirectHook;
        params?: QueryParams;
        validators?: CacheValidators;
    }

    • Optionalheaders?: Record<string, string>

    • OptionalmaxRetries?: number

      Fewer retries than the engine's for this request (never more): robots.txt asks once more at most.

    • OptionalonRedirect?: RedirectHook

      Called with each redirect target before it is requested; throw to refuse the hop.

    • Optionalparams?: QueryParams

    • Optionalvalidators?: CacheValidators

  Returns Promise<FetchResult>

head

• head(
      pathOrUrl: string,
      options?: {
          headers?: Record<string, string>;
          onRedirect?: RedirectHook;
          params?: QueryParams;
      },
  ): Promise<
      {
          headers: Record<string, string | string[] | undefined>;
          status: number;
      },
  >

  HEAD a URL: does this document exist, without downloading it.

  It is here for the case where a path carries a fact. Bayern files its Drucksachen under …/Drucksachen/Schriftliche Anfragen/19_0013327.pdf, so a 200 there is proof that the paper is a Schriftliche Anfrage and a 404 is proof that it is not — which is how that connector tells one instrument from another without guessing and without pulling half a megabyte per candidate.

  Unlike get, a 404 is returned rather than thrown: absence is the answer being asked for.

  Parameters

  • pathOrUrl: string
  • Optionaloptions: {
        headers?: Record<string, string>;
        onRedirect?: RedirectHook;
        params?: QueryParams;
    }

  Returns Promise<
      {
          headers: Record<string, string | string[] | undefined>;
          status: number;
      },
  >

post

• post(
      pathOrUrl: string,
      options?: {
          body: string;
          contentType?: string;
          headers?: Record<string, string>;
          params?: QueryParams;
      },
  ): Promise<FetchResult>

  POST a body. Only one source needs this — Thüringen's Parlamentsdatenbank drives its search from a JSON API rather than a GET form — and it goes through the same retry, redirect and rate-limiting path as everything else, so being a good citizen is not something that adapter has to remember.

  Parameters

  • pathOrUrl: string
  • Optionaloptions: {
        body: string;
        contentType?: string;
        headers?: Record<string, string>;
        params?: QueryParams;
    }

  Returns Promise<FetchResult>

raiseMinHostInterval

• raiseMinHostInterval(ms: number): void

  Go no faster than ms between two requests to any one host, from now on. Only ever raises the engine-wide interval, never lowers it: a source's politeness floor (Source.minHostIntervalMs) covers every host it reaches, which is not known in advance (an aggregator for discovery, a Land's server for the documents). sync() calls it with the source's floor. Throws OpenKaValidationError for a value the minHostIntervalMs option refuses.

  Parameters

  • ms: number

  Returns void

slowDown

• slowDown(host: string, ms: number): void

  Go no faster than ms between requests to host from now on. Only ever raises the floor: a host that asked not to be crawled and is fetched anyway is slowed to the rate the override policy names, whatever the global setting.

  Parameters

  • host: string
  • ms: number

  Returns void

url

• url(pathOrUrl: string, params?: QueryParams): string

  Resolve a path (or absolute URL) plus params against the configured base.

  Parameters

  • pathOrUrl: string
  • Optionalparams: QueryParams

  Returns string