Websites

A Website source copies the pages of a public website into your workspace. Olyvr reads the address you give, follows the links on that site, and keeps one document per page. Documentation sites, a product’s help pages and a company’s public site are the usual cases. Add as many websites as you like: each one is a source of its own, with its own settings and refresh.

Add a website

1. Open Workspace from the sidebar and choose Add data source → Website.

2. Enter the address to start from, such as https://example.com/docs. Check the address shows the page’s title, how many links it has on the site and which reader will read it, or tells you when the site’s robots.txt keeps crawlers off that page.

3. Choose how far to follow links (none, or up to five links away from the start page) and how many pages to take at most (10 to 500). Under Advanced, a path such as /docs keeps the crawl under that part of the site; the start address has to be under it. With Only this page, the sitemap’s other pages aren’t read either.

4. Start import. The source appears with its page count as the crawl goes on; pages come in breadth first, so the ones nearest the start page arrive first.

How pages are read

Olyvr uses the best reader you have:

• Firecrawl, when the Firecrawl plugin is connected. Firecrawl opens each page as a browser would, so sites built with JavaScript come in too. It runs on Firecrawl’s servers and uses your Firecrawl key.
• Olyvr’s own reader otherwise. It reads the page as the site sends it, without running its JavaScript.

The Fetch plugin isn’t used for crawls: it follows a site’s redirects on its own, which could lead it to an address on your network.

Whichever reader fetches a page, the page is taken apart (its text, title and links) in Olyvr’s Python sandbox, which can’t reach your files, your network or anything else on this computer. Olyvr checks every address the crawl asks for before any reader is asked: same site, allowed by robots.txt, public. A page Firecrawl ends up reading from another site is not kept.

What is read

• When the site has a sitemap (named in its robots.txt, or at /sitemap.xml), only the pages it lists, besides the start page. A page linked from the site but missing from its sitemap is not read, and pages the sitemap lists that nothing links to come in too, after the linked ones. A sitemap that lists no other page under the source’s path, or fewer than half of the start page’s links, is out of date or partial: it is set aside and the links lead, and Check the address says so.
• Only pages on the same site as the start address (example.com and www.example.com count as one site). Links to other sites, mail links and files Olyvr can’t read (images, archives) are skipped, and a redirect that leaves the site is not followed. A sitemap on another site is not read.
• HTML pages, and the PDF, text, Markdown, CSV and JSON files they link to.
• Only what the site’s robots.txt allows a crawler called Olyvr to read. Pages it disallows are skipped. When the site’s robots.txt can’t be read (the site is down or busy), nothing is read: the crawl tries again on the next refresh.
• Only public addresses. Nothing on this computer or your network can be crawled, and an address with a sign-in in it is refused.

Each document opens at its own address with Open original. The crawl runs again on auto-refresh and when you press Update; pages that changed are replaced, and pages no longer reached leave the source. A page that couldn’t be read this time (the site was busy) keeps its last copy; a page the site says is gone (404) leaves.

Caution

A website source reads only what the site shows to anyone. Pages behind a sign-in are not read, even when you are signed in to that site in your browser. To import those, save them as files or connect the account that owns them.

Limits

• 500 pages per source, five links deep at most
• 30 MB per page or file
• One request at a time, with a short pause between requests, so a crawl of several hundred pages takes a few minutes