Web scraping

Before using this software read article: Is web scraping perfectly legal ?

Licence: MIT

Requirements

1. JDK gte 1.8
2. Sbt gte 1.x.x
3. Docker
4. Terraform

AWS Deployment includes:

• Vpc - Networking - (ipv4, ipv6, public & private subnets)
• Private dns zone
• Mongodb instance
• Elastic search domain
• Lambda cloud formation custom resource
• AWS CodeBuild
• AWS CloudWatch (Events, Logs)
• AWS CloudFormation

More on AWS

Running

• Debugging mode:

docker-compose up -d
sbt run -jvm-debug 5005 -J-Xmx4G -Dconfig.resource=application.dev.conf

• Prod mode:

You should change prod.conf and set logging to ERROR mode if you running production mode.

docker-compose up -d
sbt runProd -J-Xmx4G -Dconfig.resource=application.dev.conf

• Docker:

docker-compose up -d
sbt docker:publishLocal
docker run -d -p 9000:9000 sphere-api-crawlers:1.0-SNAPSHOT

API

Schedule job

• HOST: http://localhost:9000
• METHOD: POST
• PATH: /crawler/v2

Multiple jobs:

[
  {
    "url": "https://typeix.github.io",
    "config": {
      "concurrency": 1,
      "throttle": 1000
    }
  },
  {
    "url": "https://en.wikipedia.org",
    "include": [
      "/wiki"
    ],
    "config": {
      "concurrency": 1,
      "throttle": 1000
    }
  },
  {
    "url": "https://en.wikipedia.org",
    "exclude": [
      "/wiki" 
    ],
    "config": {
      "concurrency": 1,
      "throttle": 1000
    }
  }
]

Kill scheduled job

• HOST: http://localhost:9000
• METHOD: DELETE
• PATH: /crawler/v2

Multiple jobs:

[
  {
    "url": "https://typeix.github.io"
  }
]

CONFIG OPTIONS

Task	Type	Description
url	String	link to crawl
include	List[String]	crawl only paths which are in include list
exclude	List[String]	crawl everything except paths in exclude list

Config - Key	Type	Description
throttle	Integer	crawling delay in ms
concurrency	Integer	number of concurrent ops
withIndexThrottle	Boolean	see index throttle formula
withStripOtherQueries	Boolean	in combination with include

Throttle formula

• withIndexThrottle - true - default false: Current pending size * throttle = time of delay If concurrency is 5 and throttle 1000, sphere will crawl 5 pages at least each second, delay is prolonged based on current pending queue, so if current pending queue is 100 sphere will crawl 5 pages each 100 seconds so if page have a lot of links and if withIndexThrottle is enabled throttle should not be number bigger than 10.

• withIndexThrottle - false is default behavior: throttle = time of delay If concurrency is 5 and throttle 1000, sphere will crawl 5 pages exactly each second.

After you crawl page

You can find statistics and info in elastic search. /crawler/_search

You can find all crawled pages in folder data/storage/.