What does WGET Do?
Once installed, the WGET command allows you to download files over the TCP/IP protocols: FTP, HTTP and HTTPS.
If you’re a Linux or Mac user, WGET is either already included in the package you’re running or it’s a trivial case of installing from whatever repository you prefer with a single command.
Unfortunately, it’s not quite that simple in Windows (although it’s still very easy!).
To run WGET you need to download, unzip and install manually.
Install WGET in Windows 10
Download the classic 32 bit version 1.14 here or, go to this Windows binaries collection at Eternally Bored here for the later versions and the faster 64 bit builds.
Here is the downloadable zip file for version 1.2 64 bit.
If you want to be able to run WGET from any directory inside the command terminal, you’ll need to learn about path variables in Windows to work out where to copy your new executable. If you follow these steps, you’ll be able to make WGET a command you can run from any directory in Command Prompt.
Run WGET from anywhere
Firstly, we need to determine where to copy WGET.exe.
After you’d downloaded wget.exe (or unpacked the associated distribution zip files) open a command terminal by typing “cmd” in the search menu:
We’re going to move wget.exe into a Windows directory that will allow WGET to be run from anywhere.
First, we need to find out which directory that should be. Type:
path
You should see something like this:
Thanks to the “Path” environment variable, we know that we need to copy wget.exe to the c:\Windows\System32 folder location.
Go ahead and copy WGET.exe to the System32 directory and restart your Command Prompt.
Restart command terminal and test WGET
If you want to test WGET is working properly, restart your terminal and type:
wget -h
If you’ve copied the file to the right place, you’ll see a help file appear with all of the available commands.
So, you should see something like this:
Now it’s time to get started.
Get started with WGET
Seeing that we’ll be working in Command Prompt, let’s create a download directory just for WGET downloads.
To create a directory, we’ll use the command md (“make directory”).
Change to the c:/ prompt and type:
md wgetdown
Then, change to your new directory and type “dir” to see the (blank) contents.
Now, you’re ready to do some downloading.
Example commands
Once you’ve got WGET installed and you’ve created a new directory, all you have to do is learn some of the finer points of WGET arguments to make sure you get what you need.
The Gnu.org WGET manual is a particularly useful resource for those inclined to really learn the details.
If you want some quick commands though, read on. I’ve listed a set of instructions to WGET to recursively mirror your site, download all the images, CSS and JavaScript, localise all of the URLs (so the site works on your local machine), and save all the pages as a .html file.
To mirror your site execute this command:
wget -r https://www.yoursite.com
To mirror the site and localise all of the urls:
wget --convert-links -r https://www.yoursite.com
To make a full offline mirror of a site:
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent https://www.yoursite.com
To mirror the site and save the files as .html:
wget --html-extension -r https://www.yoursite.com
To download all jpg images from a site:
wget -A "*.jpg" -r https://www.yoursite.com
For more filetype-specific operations, check out this useful thread on Stack.
Set a different user agent:
Some web servers are set up to deny WGET’s default user agent – for obvious, bandwidth saving reasons. You could try changing your user agent to get round this. For example, by pretending to be Googlebot:
wget --user-agent="Googlebot/2.1 (+https://www.googlebot.com/bot.html)" -r https://www.yoursite.com
Wget “spider” mode:
Wget can fetch pages without saving them which can be a useful feature in case you’re looking for broken links on a website. Remember to enable recursive mode, which allows wget to scan through the document and look for links to traverse.
wget --spider -r https://www.yoursite.com
You can also save this to a log file by adding this option:
wget --spider -r https://www.yoursite.com -o wget.log
Enjoy using this powerful tool, and I hope you’ve enjoyed my tutorial. Comments welcome!
WGET is a free tool to crawl websites and download files via the command line.
In this wget tutorial, we will learn how to install and how to use wget commands with examples.
What is Wget?
Wget is free command-line tool created by the GNU Project that is used todownload files from the internet.
- It lets you download files from the internet via FTP, HTTP or HTTPS (web pages, pdf, xml sitemaps, etc.).
- It provides recursive downloads, which means that Wget downloads the requested document, then the documents linked from that document, and then the next, etc.
- It follows the links and directory structure.
- It lets you overwrite the links with the correct domain, helping you create mirrors of websites.
What Is the Wget Command?
The wget command is a tool developed by the GNU Project to download files from the web. Wget allows you to retrieve content and files from web servers using a command-line interface. The name “wget” comes from “World Wide Web” and “get”. Wget supports downloads via FTP, SFTP, HTTP, and HTTPS protocols.
Wget is used by developers to automate file downloads.
How to Install Wget
Check if the Wget Command is installed
Open Terminal and type:
If it is installed, it will return the version.
If not, follow the next steps to download wget on either Mac or Windows.
Download and Install Wget on Mac
The recommended method to install wget on Mac is with Homebrew.
First, install Homebrew.
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"Then, install wget.
$ brew install wget
Download and Install Wget on Windows
To install and configure wget for Windows:
- Download wget for Windows and install the package.
- Copy the
wget.exefile into yourC:\Windows\System32folder. - Open the command prompt (cmd.exe) and run
wgetto see if it is installed.
Here is a quick video showing you how to download wget on windows 10.
Download and Install Wget on Linux
To install Wget on Linux Ubuntu/Debian use the apt-get command.
$ apt-get install wgetAnd verify installation with the wget command with the --version flag.
$ wget --versionDownloading Files From the Command Line (Wget Basics)
Let’s look at the wget syntax, view the basic commands structure and understand the most important options.
Wget Syntax
Wget has two arguments: [OPTION] and [URL] .
wget [OPTION]... [URL]...
- [OPTION] tells what to do with the [URL] argument provided after. It has a short and a long-form (ex:
-Vand--versionare doing the same thing). - [URL] is the file or the directory you wish to download.
- You can call many OPTIONS or URLs at once.
View WGET Arguments
To view available wget Arguments, use the wget help command:
The output will show you an exhaustive list of all the wget command parameters.
Here are the 11 best things that you can do with Wget:
- Download a single file
- Download a files to a specific directory
- Rename a downloaded files
- Define User Agent
- Extract as Googlebot
- Extract Robots.txt when it changes
- Convert links on a page
- Mirror a single page
- Extract Multiple URLs from a list
- Limit Speed
- Number of attempts
- Use Proxies
- Continue Interrupted Downloads
- Extract Entire Website
Download a single file with Wget
$ wget https://example.com/robots.txt
Download a File to a Specific Output Directory
Here replace <YOUR-PATH> by the output directory location where you want to save the file.
$ wget ‐P <YOUR-PATH> https://example.com/sitemap.xml
Rename Downloaded File when Retrieving with Wget
To output the file with a different name:
$ wget -O <YOUR-FILENAME.html> https://example.com/file.html
Define User Agent in WGET
Identify yourself. Define your user-agent.
$ wget --user-agent=Chrome https://example.com/file.html
Extract as Googlebot with Wget Command
$ wget --user-agent="Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/path
Let’s extract robots.txt only if the latest version in the server is more recent than the local copy.
First time that you extract use -S to keep a timestamps of the file.
$ wget -S https://example.com/robots.txt
Later, to check if the robots.txt file has changed, and download it if it has.
$ wget -N https://example.com/robots.txt
Wget command to Convert Links on a Page
Convert the links in the HTML so they still work in your local version. (ex: example.com/path to localhost:8000/path)
$ wget --convert-links https://example.com/path
Mirror a Single Webpage in Wget
To mirror a single web page so that it can work on your local.
$ wget -E -H -k -K -p --convert-links https://example.com/path
Add all urls in a urls.txt file.
https://example.com/1 https://example.com/2 https://example.com/3
To be a good citizen of the web, it is important not to crawl too fast by using --wait and --limit-rate.
--wait=1: Wait 1 second between extractions.--limit-rate=10K: Limit the download speed (bytes per second)
Define Number of Retry Attempts in Wget
Sometimes the internet connection fails, sometimes the attempts it blocked, sometimes the server does not respond. Define a number of attempts with the -tries function.
$ wget -tries=10 https://example.com
How to Use Proxies With Wget?
To set a proxy with Wget, we need to update the ~/.wgetrc file located at /etc/wgetrc.
You can modify the ~/.wgetrc in your favourite text editor.
$ vi ~/.wgetrc # VI $ code ~/.wgetrc # VSCode
And add these lines to the wget parameters:
use_proxy = on http_proxy = http://username:password@proxy.server.address:port/ https_proxy = http://username:password@proxy.server.address:port/
Then, by running any wget command, you’ll be using proxies.
Alternatively, you can use the -e command to run wget with proxies without changing the environment variables.
wget -e use_proxy=yes -e http_proxy=http://proxy.server.address:port/ https://example.com
How to remove the Wget proxies?
When you don’t want to use the proxies anymore, update the ~/.wgetrc to remove the lines that you added or simply use the command below to override them:
Continue Interrupted Downloads with Wget
When your retrieval process is interrupted, continue the download with restarting the whole extraction using the -c command.
$ wget -c https://example.com
Recursive mode extract a page, and follows the links on the pages to extract them as well.
This is extracting your entire site and can put extra load on your server. Be sure that you know what you do or that you involve the devs.
$ wget --recursive --page-requisites --adjust-extension --span-hosts --wait=1 --limit-rate=10K --convert-links --restrict-file-names=windows --no-clobber --domains example.com --no-parent example.com
| Command | What it does |
|---|---|
| –recursive | Follow links in the document. The maximum depth is 5. |
| –page-requisites | Get all assets (CSS/JS/images) |
| –adjust-extension | Save files with .html at the end. |
| –span-hosts | Include necessary assets from offsite as well. |
| –wait=1 | Wait 1 second between extractions. |
| –limit-rate=10K | Limit the download speed (bytes per second) |
| –convert-links | Convert the links in the HTML so they still work in your local version. |
| –restrict-file-names=windows | Modify filenames to work in Windows. |
| –no-clobber | Overwrite existing files. |
| –domains example.com | Do not follow links outside this domain. |
| –no-parent | Do not ever ascend to the parent directory when retrieving recursively |
| –level | Specify the depth of crawling. |
$ wget --spider -r https://example.com -o wget.log
Wget VS Curl
Wget’s strength compared to curl is its ability to download recursively. This means that it will download a document, then follow the links and then download those documents as well.
Use Wget With Python
Wget is strictly command line, but there is a package that you can import the wget package that mimics wget.
import wget url = 'http://www.jcchouinard.com/robots.txt' filename = wget.download(url) filename
Debugging: What to Do When Wget is Not Working
Wget Command Not Found
If you get the -bash: wget: command not found error on Mac, Linux or Windows, it means that the wget GNU is either not installed or does not work properly.
Go back and make sure that you installed wget properly.
Wget is not recognized as an internal or external command
If you get the following error
'wget' is not recognized as an internal or external command, operable program or batch file
It is more than likely that the wget package was not installed on Windows. Fix the error by installing wget first and then start over using the command.
Alternatives to Wget on Mac and Windows
You can use cURL as an alternative of Wget command line tool. It also has to be installed on Mac, Linux and Windows.
Wget for Web Scraping
By allowing you to download files from the Internet, the wget command-line tool is incredibly useful in web scraping. It has a set of useful features that make web scraping easy:
- Batch Downloading:
wgetallows you to download multiple files or web pages in a single command. - Recursive Downloading: the
--recursiveflag inwgetallows you to follow links and download an entire website - Retries:
wgetis designed to handle unstable network connections and interruptions and retry failed extractions - Command-line options: Options are available to improve scraping capabilities (download speed, User-Agent headers, cookies for authentication, etc.).
- Header and User-Agent Spoofing: To avoid being blocked by websites when web scraping,
wgetallows you to change the User-Agent header to make your requests appear more regular users. - Limiting Server Load: By using the
--waitand--limit-rateoptions, you can control the speed at whichwgetfetches data.
About Wget
| Wget was developed by | Hrvoje Nikšić |
| Wget is Maintained by | Tim Rühsen and al. |
| Wget Supported Protocols | HTTP(S), FTP(S) |
| Wget was Created In | January 1996 |
| Installing Wget | brew install wget |
| Wget Command | wget [option]…[URL]… |
Wget FAQs
What is Wget Used For?
Wget is used to download files from the Internet without the use of a browser. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.
How Does Wget Work?
Wget is non-interactive and allows to download files from the internet in the background without the need of a browser or user interface. It works by following links to create local versions of remote web sites, while respecting robots.txt.
What is the Difference Between Wget and cURL?
Both Wget and cURL are command-line utilities that allow file transfer from the internet. Although, Curl generally offers more features than Wget, wget provide features such as recursive downloads.
Can you Use Wget With Python?
Yes, you can run wget get in Python by installing the wget library with $pip install wget
Does Wget Respect Robots.txt?
Yes, Wget respects the Robot Exclusion Standard (/robots.txt)
Is Wget Free?
Yes, GNU Wget is free software that everyone can use, redistribute and/or modify under the terms of the GNU General Public License
What is recursive download?
Recursive download, or recursive retrieval, is the capacity of downloading documents, follow the links within them and finally downloading those documents until all linked documents are downloaded, or the maximum depth specified is reached.
How to specify download location in Wget?
Use the -P or –directory-prefix=PREFIX. Example: $ wget -P /path <url>
Conclusion
This is it.
You now know how to install and use Wget in your command-line.
SEO Strategist at Tripadvisor, ex- Seek (Melbourne, Australia). Specialized in technical SEO. In a quest to programmatic SEO for large organizations through the use of Python, R and machine learning.
Most users will download files onto their PC using their web browser. There’s a problem with this method, however—it’s not particularly efficient. If you need to pause your download, or if you’ve lost your connection, you’ll probably need to start your download again from scratch. You may also be working with Python or other code at the command line and want to download directly from the command prompt.
That’s where tools like Wget come in. This command line tool has a number of useful features, with support for recursive downloads and download resumption that allows you to download single files (or entire websites) in one go.
Wget is popular on Linux and other Unix-based operating systems, but it’s also available for Windows users. Below, we’ll explain how to install and use Wget to download any content you want online from your Windows command line.
Installing GNU Wget on Windows
Wget (in name, at least) is available on Windows 10 and 11 via the PowerShell terminal. However, this version of Wget isn’t the same as the GNU Wget tool that you’d use on a Linux PC. Instead, this version is simply an alias for a PowerShell command called Invoke-WebRequest.
Invoke-WebRequest is Wget-like in what it does, but it’s a completely different tool that’s much more difficult to use and understand. Instead, you’ll be better served by installing Wget for Windows, a compiled version of the same tool available for Linux users, using the steps below.
1. Download the Wget for Windows setup file from the Wget website. You’ll need to do this using your web browser.
2. Run the Wget for Windows installer file. Once the Wget setup file has finished downloading, run the setup file and follow the on-screen instructions to complete the installation.
3. Update the Wget.exe file (optional). The Wget installer is packaged with a fairly old version of the Wget binary. If you run into difficulties downloading files because of SSL certificate errors, you should download the latest wget.exe for your architecture from this website and save it to your Wget installation directory (typically C:\Program Files (x86)\GnuWin32\bin). This step is optional, but highly recommended.
4. Open the Start menu, search for environment variables, and click Open. Once the installation is finished, use the search tool in the Start menu to search for environment variables, then click Open. You’ll need to do this to allow you to use the ‘wget’ command from the command line without referencing its location every time you wish to run it.
5. Click Environment Variables in the System Properties window.
6. Select Path and click Edit under System or User variables.
7. Click the New button and type in the directory for the Wget for Windows binary (.exe) file. By default, this should be C:\Program Files (x86)\GnuWin32\bin.
8. Save your changes. When you’re finished, click OK in each menu and exit System Properties.
9. Open the Start menu, type cmd, and press Open. This will launch a new command prompt window. You can also use the newer Terminal app, as long as you switch to using a command prompt shell.
10. Type wget —version and press Enter. If Wget was installed correctly, you should see the GNU Wget version returned in the command prompt window.
If you want to run Wget from a PowerShell terminal instead, you’ll need to run the file from its installation directory directly (eg. C:\Program Files (x86)\GnuWin32\bin\wget.exe).
Downloading Files with Wget
Once you’ve installed GNU Wget and you’ve configured the environment variables to be able to launch it correctly, you’ll be able to use it to start downloading files and webpages.
We’ve used an example domain and file path in our examples below. You’ll need to replace this with the correct path to the file (or files) that you want to download.
- Type wget -h to see a full list of commands. This will give you the full list of options that you can use with Wget.
wget -h
- Download a single file using wget <url>. Replace <url> with the path to a file on an HTTP, HTTPS, or FTP server. You can also refer to a website domain name or web page directly to download that specific page (without any of its other content).
wget example.com
- Save with a different filename using -O. Using the -O option, you’ll be able to save the file with a different filename. For example, wget -O <filename> <url>, where <filename> is the filename you’ve chosen.
wget -O example.html example.com
- Save to a different directory using -P. If you want to save to another directory than the one you’re currently in, use the -P option. For example, wget -P <path> <url>.
wget -P C:\folder example.com
- Use —continue or -c to resume files. If you want to resume a partial download, use the -c option to resume it, as long as you’re in the same directory. For example, wget -c <url>.
wget -c example.com
- Download multiple files in sequence. If you want to download several files, add each URL to your Wget command. For example, wget <url1> <url2> etc.
wget example.com tomshardware.com
- Download multiple files using a text file with -i. Using the -i option, you can refer to a text file that contains a list of URLs to download a large number of files. Assuming that each URL is on a new line, Wget will download the content from each URL in sequence. For example, wget -i <file.txt> <url>.
wget -i urls.txt
- Limit download speeds using —limit-rate. If you want to limit your bandwidth usage, you can cap the download speeds using the —limit-rate option. For example, wget —limit-rate=1M <url> would limit it to 1 megabyte per second download speeds, while wget —limit-rate=10K <url> would limit it to 10 kilobytes per second.
wget —limit-rate=10K example.com
- Use -w or –wait to set a pause period after each download. If you’re downloading multiple files, using -w can help to spread the requests you make and help to limit any chance that your downloads are blocked. For example, wget -w 10 <url1> <url2> for a 10 second wait.
wget -w 10 example.com tomshardware.com
- Set a retry limit using -t or —tries. If a download fails, wget will use the -t value to determine how many times it’ll attempt it again before it stops. The default value is 20 retries. If the file is missing, or if the connection is refused, then this value is ignored and Wget will terminate immediately.
wget -t 5 example.com
- Save a log using -o or -a. You can save your log data to a text file using -o (to always create a new log file) or -a (to append to an existing file). For example, wget -o <file.txt> <url>.
- Bypass SSL errors using —no-check-certificate. If you’re having trouble downloading from a web server with an SSL certificate and you’ve already updated your Wget installation, bypass the SSL certificate check completely using —no-check-certificate to allow the download (in most cases). You should only do this for downloads from locations that you completely trust. For example, wget —no-check-certificate example.com.
wget —no-check-certificate https://example.com
Make sure to use the wget -h or wget —help command to view the full list of options that are available to you. If you run into trouble with Wget, make sure to limit the number of retries you make and set a wait limit for each download you attempt.
Using Wget for Recursive Downloads
One of Wget’s most useful features is the ability to download recursively. Instead of only downloading a single file, it’ll instead try to download an entire directory of related files.
For instance, if you specify a web page, it’ll download the content attached to that page (such as images). Depending on the recursive depth you choose, it can also download any pages that are linked to it, as well as the content on those pages, any pages that are linked on those pages, and so on.
Theoretically, Wget can run with an infinite depth level, meaning it’ll never stop trying to go further and deeper with the content it downloads. However, from a practical point of view, you may find that most web servers will block this level of scraping, so you’ll need to tread carefully.
- Type wget -r or wget —recursive to download recursively. By default, the depth level is five. For example, wget -r <url>.
wget -r tomshardware.com
- Use -l or –level to set a custom depth level. For example, wget -r -l 10 <url>. Use wget -r -l inf <url> for an infinite depth level.
wget -r -l 10 tomshardware.com
- Use -k to convert links to local file URLs. If you’re scraping a website, Wget will automatically convert any links in HTML to point instead to the offline copy that you’ve downloaded. For example, wget -r -k <url>.
wget -r -k tomshardware.com
- Use -p or —page-requisites to download all page content. If you want a website to fully download so that all of the images, CSS, and other page content is available offline, use the -p or —page-requisites options. For example, wget -r -p <url>.
wget -r -p tomshardware.com
For a full list of options, make sure to use the wget —h command. You should also take care to respect any website that you’re actively downloading from and do your best to limit server loads using wait, retry, and depth limits.
If you run into difficulties with downloads because of SSL certificate errors, don’t forget to update your Wget binary file (wget.exe) with the latest version.
Join the experts who read Tom’s Hardware for the inside track on enthusiast PC tech news — and have for over 25 years. We’ll send breaking news and in-depth reviews of CPUs, GPUs, AI, maker hardware and more straight to your inbox.
Most Popular
What is wget?
Wget is a free GNU command-line utility tool used to download files from the internet. It retrieves files using HTTP, HTTPS, and FTP protocols.
It serves as a tool to sustain unstable and slow network connections. If a network problem occurs during a download, this helpful software can resume retrieving the files without starting from scratch.
Another important aspect is its capability of recursive downloads, with which it mirrors websites. It transfers parts of a website by following links and directory structure, thus creating local versions of webpages.
The wget command is also highly flexible and can be used in terminals, scripts, and cron jobs. During the download, the user does not have to be active nor logged in. As wget is non-interactive, it can independently run in the background.
Read this article to learn how to use some of the most common wget commands.
How to Check if wget is Installed?
Most likely, the wget package is already on your system as it now comes pre-installed on most Linux distributions.
To check, open the terminal window and type in:
wgetIf you have the wget software, the output tells you that the wget command is missing a URL, as shown in the image below:
wget Command Not Found
If the output displays wget command not found you need to download and install the tool manually. Below you will find the installation instructions for Ubuntu/Debian, CentOS, and Windows.
How to Install wget on Ubuntu/Debian?
To install wget on Ubuntu or Debian releases, use the command:
sudo apt-get install wgetHow to Install wget on CentOS/Fedora?
To install wget on CentOS or Fedora, type the following command:
sudo yum install wgetHow to Install wget on Windows?
To install and configure wget for Windows:
- Download wget for Windows and install the package.
- Add the wget bin path to environment variables (optional). Configuring this removes the need for full paths, and makes it a lot easier to run wget from the command prompt:
- Open the Start menu and search for “environment.”
- Select Edit the system environment variables.
- Select the Advanced tab and click the Environment Variables button.
- Select the Path variable under System Variables.
- Click Edit.
- In the Variable value field add the path to the wget bin directory preceded by a semicolon (;). If installed in the default path, add
C:Program Files (x86)GnuWin32bin.
- Open the command prompt (cmd.exe) and start running
wgetcommands.
Introduction to wget Syntax
The wget syntax has the following pattern:
wget [option][URL]Each [option] has its long and short form which are conveniently interchangeable. This attribute specifies what to do with the URL that follows.
[URL] is the address of the file or directory you wish to download.
Download File from Web
To download a file from the web use:
wget [URL]For example, to install Tomcat 9, first you need to download the package with wget using the command:
wget http://apache.cs.utah.edu/tomcat/tomcat-9/v9.0.20/bin/apache-tomcat-9.0.20.tar.gzDownload File and Save Under Specific Name
To download a file and save it under a specified name run:
wget -O [file_name] [URL]The wget command allows you to rename files prior to downloading them on your computer.
For instance, you may want to install Terraform. To download the package and rename it terraform.zip use the following command:
wget -O terraform.zip https://releases.hashicorp.com/terraform/0.12.2/terraform_0.12.2_linux_amd64.zipDownload File to Specific Directory
By default wget downloads a file in the directory the user is in. To save the file in a different location, add the -P option:
wget -P [wanted_directory] [URL]For example, while installing Git on Ubuntu, you can download the package in the /temp directory with the command:
wget -P /temp https://github.com/git/git/archive/master.zipSet Download Speed
You can set the download speed when downloading a big file, so it does not use the full available bandwidth. The download speed is defined in kilobytes (k) and megabytes (m). Use the command:
wget --limit-rate [wanted_speed] [URL]For example, if you are installing NVIDIA TESLA drivers on Linux and want to limit the download speed to 1 megabyte, would use the command:
wget --limit-rate 1m http://us.download.nvidia.com/tesla/396.37/nvidia-diag-driver-local-repo-ubuntu1710-396.37_1.0-1_amd64.deb Continue Download After Interruption
Instead of having to start from scratch, wget can resume downloading where it stopped before the interruption. This is a useful feature if there is a lost of connection while downloading a file.
wget -c [URL]For instance, you may want to install a Mumble Server on Linux and suddenly lose internet connection while downloading the installation file. To continue downloading, type in the command:
wget -c https://github.com/mumble-voip/mumble/releases/download/1.2.19/murmur-static_x86-1.2.19.tar.bz2Download Multiple Files
wget allows downloading multiple files at the same time using the command:
wget -i [file_name]To do so, follow the steps outlined below:
1. First, create and open a file under the name MultipleDownloads.txt (or a name of your choice), using a text editor. In this case, we used Nano:
nano MultipleDownloads.txt2. Once in the editor, add the URLs of the packages you want to download, one per line.
3. Save and exit the file.
4. Run the following wget command in the terminal window:
wget -i MultipleDownloads.txtThis prompts wget to download from each URL in the text file.
Download Web page (Mirror Web page)
With wget you can download an entire website from the internet, using the -m option. It prompts wget to create a mirror of the specified webpage. The basic command for doing so is:
wget -m [URL]For example:
wget -m <a href="https://phoenixnap.com/" target="_blank" rel="noreferrer noopener">https://phoenixnap.com</a>Download via FTP
To download via FTP, type in the username and password of the FTP server, followed by the ftp address:
wget --ftp-user=[ftp_username] --ftp-password=[ftp_password] ftp://...For instance:
wget --ftp-user=sofiftp --ftp-password=TopSecretPassword ftp://123.456.7890Download in Background
You can download in the background, a practical feature when dealing with a large file:
wget -b [URL]You can check the status of the download with the command:
tail -f wget -logTo download the RPM package manager in the background, type:
wget -b http://some_website/sample_file.rpmIncrease Retry Attempts
You can set how many times wget attempts to download a file after being interrupted by a bad network with:
wget --tries=[number_of_tries] [URL]By default, the number of retry attempts is set to 20.
You can also set the number to infinity with the values 0 or inf, as in the following example:
wget --tries=inf http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repoSkip Certificate Check
By default, wget checks whether the server has a valid SSL/TLS certificate. If it does not identify an authentic certificate, it refuses to download.
The --no-check-certificate option is used to avoid certificate authorities checking for a server certificate. However, utilize it only if you are sure of the website’s credibility or are not worried about security issues it may cause.
wget --no-check-certificate [URL]If http://enteratonerisk.com has an untrusted certificate, but will not harm the system, you can download it with:
wget --no-check-certificate http://enteratonerisk.comChange User Agent
When downloading a webpage, wget essentially emulates a browser. In some cases, the output might say you don’t have permission to access the server, or that the connection is forbidden. This may be due to a website blocking client browsers that have a specific “User-Agent.”
“User-Agent” is a header field that the browser sends to the server it wants to access. Therefore, to download from a server that is refusing to connect, try to modify the user agent.
Find a database of all user agents online, search for the one you need and run the command:
wget --user-agent="User Agent Here" "[URL]"or
wget -U "User Agent Here" "[URL]"For example, to emulate Chrome (version 74), you would change the user agent with the command:
wget --user-agent=" Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" "https://phoenixnap.com"Note: Make sure you always download from a trusted source because wget can instruct a download of a script from a malicious source. We outline this and many other dangerous commands in our article 14 Dangerous Linux Terminal Commands.
Conclusion
This article sums up why wget is such a powerful tool for downloading files over the internet. It also serves as a good reference for beginners with its list of 12 essential wget commands and examples.
Have you ever lost an essential asset at a crucial moment on your website? It is frightening to even think about it, isn’t it? Perhaps if you have used Linux, then you might have heard of WGET. Yay! WGET is also available for Windows. Thank GNU for coming up with a compatible version of WGET for Windows 10. From the starting of this article, you will understand how to download and install WGET for Windows 10. Also, you will learn how to use WGET command with examples provided for improving your understanding. Keep on reading to gain comprehensive knowledge about WGET.
Table of Contents
WGET has been available for Linux users for a long time. Learners can even refer to the different articles and guides available on the web to better their comprehension. But when it comes to the Windows version, even the scarce amount of articles cannot give you the information you need. However, do not lose hope. From this point on, you will know things about WGET on Windows that you’ve never heard or read. So why not start with answering the most asked question: What is WGET?
What is WGET?
WGET is a free tool to download files and crawl websites using the command line. Before learning to download WGET and how to use WGET command with examples, check out the points given below to know more about its features:
- WGET can retrieve contents from web pages.
- It even allows you to download from any web pages (FTP, HTTP) in any format (PDF, XML).
- It’s known to provide operational anonymity to keep users’ location or activities a secret.
- WGET works perfectly in bad network conditions too.
- Apart from these features, WGET can overwrite correct domain name in links.
- Moreover, it can perform recursive downloads, where the link present in the downloading document gets automatically downloaded further.
Also Read: What is a Command Line Interpreter?
How to Install WGET for Windows 10
Using WGET for Windows 10 will seamlessly download and extract information from any webpage by using commands. Implement the steps below to download GnuWin first to create a Windows environment and use WGET.
1. Download GnuWin WGET from here. The download process will start and finish automatically in a few seconds.
2. Click the downloaded WGET setup.exe folder to open it, as shown below.
3. Click Yes when prompted.
Note: Make sure to close already running applications before starting WGET installation.
4. To install WGET for Windows, click on Next, as depicted below.
5. Select the I accept the agreement radio button and click the Next option as highlighted below to continue the installation process.
6. Click on Browse to select the Destination Location. Choose a folder where you want to set up WGET and click on Next as shown.
Note: It is recommended to use default path: C:\Program Files (x86)\GnuWin32\bin
7. Select the components you want to install among Binaries and Documentation. Click Next to proceed further.
8. Here, click on Browse and select a Destination Folder to create the program shortcut in the start menu. Then, click on Next.
Note: If you don’t want to create a shortcut, select the checkbox Don’t create a Start Menu folder.
9. Select the Additional icons you need and click Next as illustrated below:
- Create document shortcuts in Start Menu folder
- Download Sources
10. Scroll down the review tab and check if all the desired options are present. Click Install to begin installing WGET on your system.
Note: You can select Back to change any setting you need to modify.
11. Wait for the WGET installation process to complete in your Windows 10 desktop/laptop and click Finish to exit the setup.
Also Read: Fix error 1500 Another Installation is in Progress
How to Use WGET for Windows 10
There are two ways to use WGET on your system.
- First, it is accessed directly via Command Prompt using a single path.
- Second, you need to manually go to the directory page where the app is present, and then proceed to the command prompt procedure.
You can implement any of these methods to use WGET on your system at your convenience. So, now you will see both methods in action below:
Method 1: Add CD to WGET Path
Once you install WGET for Windows 10, ensure that the wget.exe folder exists in the path given in this method.
Note: If not, copy and paste the respective folder in the location shown in the below File Explorer navigation, as you are about to use only this path.
1. Press Windows + E keys simultaneously to open File Explorer.
2. Navigate to C:\Program Files (x86)\GnuWin32\bin and copy the location.
3. Type Command Prompt in the Windows search bar and click Open to launch it, as shown below.
4. Type cd C:\Program Files (x86)\GnuWin32\bin and press Enter.
5. Then type wget and hit Enter. If the command is executed, it shows that WGET is running properly.
6A. Type the required WGET for Windows 10 command to download any file in the given format:
wget [OPTION] … [URL]…
6B. Or, view help from WGET by typing ‘wget –help’
Also Read: Fix File Explorer Not Responding in Windows 10
Method 2: Add Environment Variable
Once you download WGET for Windows 10, you must change the environmental variable set in the system settings. This is a long process. Follow the steps below carefully to set up WGET for Windows 10:
1. Press the Windows key from the keyboard and type Control Panel. Click the Open option to launch it.
2. Choose View by: Large icons from the top right corner. Click on the System option.
3. Scroll down and click on the Advanced system settings as illustrated below.
4. In the Advanced tab, click the Environment Variables button as shown.
5. Environmental Variables pop-up window will appear. Select the Path under the User variables for GREEN category and click on Edit as depicted below.
6. In the Edit environmental variable window, select the Browse option.
7. Select C:\Program Files (x86)\GnuWin32\bin path and click OK as shown.
8. Then, click OK as shown.
9. Again, click OK in the System Properties window.
10. Now, open Command Prompt as explained in Method 1.
11. Type wget and press Enter on the keyboard. Once the Environment Variable is set, you will be able to run WGET regardless of any folder you are currently in.
Also Read: How to Fix Git Merge Error
How to Check the Functioning of WGET App and Commands
To check if you can access WGET for Windows 10 regardless of any folder or directory that is present, follow the steps given below:
1. Type Command Prompt on the start menu search bar and click Open to launch it.
2. Type wget and press Enter to check whether the application runs.
Note: If you receive unrecognized command error, you have downloaded the WGET Setup from the wrong source. So, ensure to use only the link given above for the download process and try again.
3A. If the test worked, then it’s great. You have done a good job!
3B. If it is not working, do not worry. Check if you have followed the steps in an orderly manner, from downloading GnuWin, installing WGET Setup, then adding WGET environment variable. Try once again and restart the command line.
Also Read: Fix ERR_EMPTY_RESPONSE in Google Chrome
How to Use WGET Command with Examples
There are thousands of WGET commands to perform specific tasks. Run WGET for Windows 10 using any of the methods given above and type any of the commands listed below in Command Prompt.
Note: You need to enter the command correctly to obtain desired results.
1. To get list of commands, use the command wget -h.
2. To download a single file, use the command wget [URL]. For instance, enter the command wget https://wordpress.org/latest.zip. Here, WordPress is chosen as an example.
3. To download to a specific folder, use the command wget ‐P [wanted directory] [URL]. For instance, wget -P /temp https://github.com/git/git/archive/master.zip.
4. To resume an interrupted download, use the command wget ‐c [URL]. For instance, wget -c https://wordpress.org/latest.zip.
5. To set the download speed, use the command wget ‐‐limit-rate [wanted speed] [URL]. For instance, wget –limit-rate 1m https://wordpress.org/latest.zip.
6. To mirror a single webpage, use the command wget -m [URL]. For instance, wget -m https://cereal.guru/.
7. To know the version of installed WGET, then use the command wget -V.
8. To download and save under a specific name, use the command wget -o [file name] [URL]. For instance, wget -o file.docx https://cereal.guru/.
9. To download a web page in the background, use the command wget -b [URL]. For instance, wget -b https://cereal.guru/.
10. To download multiple URLs, use the command wget -i [File name]. For instance, wget -i URL.txt. Before executing this command, place all the URLs in one file and include that file name in the command.
11. To download via FTP, use the command wget –ftp-user=[ftp_username] –ftp-password=[ftp_password] ftp://… For instance, wget –ftp-user=sofiftp –ftp-password=TopSecretPassword ftp://123.456.7890.
12. To change the number of retry attempts, use the command wget –tries=[number_of_tries] [URL]. For instance, wget –tries=inf https://cereal.guru/. In this example, inf denotes infinity. The default number of retries is 20.
This is how to use WGET command with examples. Click here to know about more commands and their uses.
Recommended:
- Fix Windows 10 Update Error 0x80070103
- How to Enable Telnet in Windows 10
- Complete List of Windows 11 Run Commands
- Discord Commands List
We hope that this guide was helpful and you could learn how to download, install & use WGET for Windows 10. If you have any queries or suggestions, feel free to drop them in the comments section below.