Function robotsGate

robotsGate(
    engine: FetchEngine,
    options: {
        ignoreRobots?: boolean;
        origin: string;
        path: string;
        policy?: RobotsPolicy;
    },
): Promise<RobotsVerdict>
A connector asking about its document server before it discovers anything. With the run's policy (DiscoverOptions.robots) the answer is the one every later document check of the run gets; without one it is a one-shot policy of its own.
Parameters
- engine: FetchEngine
- options: { ignoreRobots?: boolean; origin: string; path: string; policy?: RobotsPolicy }
Returns Promise<RobotsVerdict>
- Defined in packages/lib-source/dist/src/base.d.ts:352