Main Settings (Web Content Crawler)

To learn about content crawlers and content Web services, click here.

To specify the location you want to crawl and the destination folder and security for documents imported by this content crawler:

  1. In the URL to crawl box, type the URL to the site from which you want to import content.

  2. In the Crawl radius drop-down list, specify the maximum number of links away from the target page that you want to crawl. For example, if you select 1, this content crawler attempts to import every page directly linked to the target page; if you select 2, this content crawler attempts to import every page directly linked to the target page, and every page directly linked to those linked pages.

  3. By default, this content crawler creates a link to the URL you entered in step 1. If you do not want to create a link to this page, clear the Import the target page check box. For example, if you crawl the results of a search, you would not want to import the target page (the search results page); you would want to import each linked page (each result).

  4. Specify into which folders you want to import content. The content crawler attempts to import a link to every document it finds into the most subordinate subfolder within the destination folder that allows the link to pass. Click here for a flow chart showing how the content crawler determines into which folders it will import content.

  5. To require that documents pass the filters of destination folders before the documents are imported into those folders, select Apply Filter of Destination Folder. By default, documents do not need to pass the filters of destination folders, so all documents will be imported into all destination folders.

  6. To accept all imported documents into the portal and make them immediately available to users, select Automatically approve imported documents. By default, documents require approval. This means that before the link to the imported document is available to users, it must be approved by a portal administrator with at least Edit access to the destination folder.

  7. Under Document Access Privileges, you can perform the following actions to grant users and groups access to the content imported by this content crawler:


  1. Click Administration.
  2. Open the Content Crawler Editor: