wget

Ignore robots.txt

wget -e robots=off http://example.com/