Rsync from linux to windows

Wondering how to perform Rsync from Linux to Windows share? We can help you.

We help some of our customers with Rsync to copy files from one location to another. In the course it was found that it is fast and can also be used for mirroring data and incremental backups.

As part of our Server Management Services, we assist our customers with several Rsync queries.

Today, let us see how to transfer Rsync from Linux to Windows share.

What is Rsync?

A fast and versatile command-line utility, Rsync (Remote Sync) helps to synchronize files and directories between two locations over a remote shell, or from/to a remote Rsync daemon.

In addition, it provides fast incremental file transfer by transferring only the differences between the source and the destination.

We can use Rsync for mirroring data, incremental backups, copying files between systems. It acts as a replacement for scp, sftp, and cp commands.

Advantages and features of the Rsync command include:

• Rsync efficiently copies and sync files to or from a remote system.
• Supports copying links, devices, owners, groups, and permissions.
• It’s faster than SCP (Secure Copy) since it allows the transfer of just the differences between two sets of files.
• It consumes less bandwidth using the compression and decompression method while sending and receiving data.

Transfer Rsync from Linux to Windows share

Initially, we make a mount point:

mkdir /mnt/share

Then, we mount the smb share:

mount -t cifs -o username=domainusername //ip_add/ShareFolder /mnt/share

It will prompt for a password. It is always a good practice not to provide password within commands.

In addition, if we run the following command we can verify if it has mount:

mount
//ip_add/ShareFolder on /mnt/share type cifs (rw)

Similarly, we can mount it after every boot.

In order to do that we need to add it to /etc/fstab:

//ip_add/ShareFolder /mnt/share smbfs username=domainusername,password=P@S5W0rd 0 0

Now we can Rsync data to it.

For example, to rsync bob’s home directory into a dir called ‘homedir’ we run:

rsync -avz /home/bob/ /mnt/share/homedir/

[Stuck with the transfer? We are here to help you]

Conclusion

To conclude, the transfer of Rsync from Linux to Windows share can be done in a few steps. Today, we saw the best fit suggested by our Support Engineers.

var google_conversion_label = «owonCMyG5nEQ0aD71QM»;

For more information, see rsync’s github link

1. Use the remote connection desktop provided by windows

1. win + R opens the operation interface

2. Enter mstsc to open the remote connection desktop

3. Enter the address, user name and password of the computer to be connected
   

4. When connecting to remote windows, the following error occurs:
   

5. win + R opens the operation interface

6. Enter gpedit MSc open policy

7. Find the path: «computer configuration» — > «management template» — > «system» — > «credential allocation» — > «encrypted database correction»

8. Edit «encrypted database correction», as shown in the following figure
   

9. Just connect to the remote desktop again

2. windows Server Deployment

Because the synchronization target here is windows, we need to deploy the server on windows

1.1 download

Because the latest version of windows server has been charged, we use the latest free version 4.1.0 here, Click here to download

After downloading, unzip it

1.2 installation

1. Click cwRsyncServer_4.1.0_Installer.exe to install
2. Select Save folder
3. User name and password are set here: user name is customized and password is customized (if the installed computer has a login password, it must be a login password)
   
4. Click Install to complete the installation

1.3 configuration

1. Edit ICW / rsyncd Conf, so the contents of the file are as follows:

uid = 0
gid = 0
use chroot = false
strict modes = false
hosts allow = *
log file = rsyncd.log

# Module definitions
# Remember cygwin naming conventions : c:\work becomes /cygwin/c/work
#
# [test]
# path = /cygdrive/c/work
# read only = false
# transfer logging = yes



[rsync_linux246]
path = /cygdrive/d/hehuan_test/rsync_workspace/rsync_linux246/
ignore errors
read only = no
list = no
hosts allow = 192.168.*.0/255.255.255.0
auth users = apache
secrets file = /cygdrive/d/hehuan_test/rsync_workspace/rsyncd.password


[rsync_windows242]
path = /cygdrive/d/hehuan_test/rsync_workspace/rsync_windows242/
ignore errors
read only = no
list = no
hosts allow = 192.168.*.0/255.255.255.0
auth users = apache
secrets file = /cygdrive/d/hehuan_test/rsync_workspace/rsyncd.password

1. New directory: Rsync_ workspace\rsync_ Linux 246 and rsync_workspace\rsync_windows242

2. New file: rsync_workspace\rsyncd.password, as follows:

apache:apache

1. Set Rsync_ Linux 246 and Rsync_ The permissions of the windows242 directory are Rsync_ Take Linux 246 as an example: right click rsync_linux246 directory, select Properties
   
2. Modify rsyncd Password permission is 600

D:\hehuan_test>
D:\hehuan_test>cwRsyncServer_4.1.0\ICW\bin\chmod 600 rsync_workspace\rsyncd.password
cygwin warning:
  MS-DOS style path detected: rsync_workspace\rsyncd.password
  Preferred POSIX equivalent is: rsync_workspace/rsyncd.password
  CYGWIN environment variable option "nodosfilewarning" turns off this warning.
  Consult the user's guide for more details about POSIX paths:
    http://cygwin.com/cygwin-ug-net/using.html#using-pathnames

D:\hehuan_test>

1.4 start rsync service

1. Open services as follows msc
   
2. The setting results are as follows:
   

3. Synchronize Linux to windows

3.1 preparation

1. Connection test

[root@node ~]#
[root@node ~]# telnet 192.168.*.* 873
Trying 192.168.*.*...
Connected to 192.168.*.*.
Escape character is '^]'.

The above indicates that the connection is successful; If the connection fails, it may be a firewall problem. Close the firewalls on both sides or open port 873

1. Install rsync

[root@node ~]#
[root@node ~]# yum install -y rsync.x86_64
[root@node ~]#

1. Create password file

[root@node ~]# 
[root@node ~]# cat /etc/rsyncd.password 
apache
[root@node ~]# 
[root@node ~]# chmod 600 /etc/rsyncd.password 
[root@node ~]#

3.2 synchronization

[root@node ~]#
[root@node ~]# rsync -avz /data/minio/hnjgzd/uploadFiles/ apache@192.168.*.*::rsync_linux246 --delete --progress --include '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' --include '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' --include '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' --exclude '/*' --password-file=/etc/rsyncd.password
sending incremental file list
./
......ellipsis......
sent 160,824,910 bytes  received 10,715 bytes  7,480,726.74 bytes/sec
total size is 165,640,562  speedup is 1.03
[root@node ~]#

3.3 real time synchronization with inotifywait

1. Installation of inotifywait

[root@node ~]# 
[root@node ~]# yum install -y epel-release.noarch
[root@node ~]#
[root@node ~]# yum install -y inotify-tools
[root@node ~]#

1. rsync.sh synchronization script writing

[root@node ~]# 
[root@node ~]# cat rsync.sh 
#!/usr/bin/env bash

source_dir=/data/minio/hnjgzd/uploadFiles/

/usr/bin/inotifywait -mrq --timefmt '%y-%m-%d %H:%M' --format '%T %w %f %e' -e create,attrib,modify,delete ${source_dir} | while read date time dir file type
do

/usr/bin/rsync -avz ${source_dir} apache@192.168.*.*::rsync_linux246 --delete --progress --include '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' --include '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' --include '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' --exclude '/*' --password-file=/etc/rsyncd.password

echo "time: ${date} ${time}, file: ${dir}${file}, Change type: ${type}, Synchronization completed" >> /root/rsync_file.log

done
[root@node ~]#

1. Start of synchronization script

[root@node ~]# 
[root@node ~]# nohup sh /root/rsync.sh >> /root/rsync.log 2>&1 &
[root@node ~]#

1. Trigger of file synchronization initialization

[root@node uploadFiles]#
[root@node uploadFiles]# pwd
/data/minio/hnjgzd/uploadFiles
[root@node uploadFiles]#
[root@node uploadFiles]# touch test.txt
[root@node uploadFiles]#
[root@node uploadFiles]# rm test.txt -f
[root@node uploadFiles]#

4. Synchronize windows to windows

4.1 download and decompression

1. download

Client Download link

The installation file location is shown in the figure below
2. Decompression
After decompression, you can use it directly

4.2 connection test

C:\Users\dell>
C:\Users\dell>telnet 192.168.*.* 873
C:\Users\dell>

If the connection fails, it may be a firewall problem. Close the firewalls on both sides or open port 873

4.3 password file

1. Create a password file in the directory D:\hehuan_test\rsync_workspace\rsyncd_client.password
2. rsyncd_ client. The content of the password file is apache

4.4 synchronization script

Script path: D:\hehuan_test\rsync_workspace\rsync.bat

The script is as follows:

%date% %time%

d:

d:\hehuan_test\cwRsyncClient_6.2.1\bin\rsync -avz data/hnjgzd/uploadFiles/ apache@192.168.*.*::rsync_windows242 --delete --progress --include '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' --include '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' --include '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' --exclude '/*' --password-file=d:\hehuan_test\rsync_workspace\rsyncd_client.password

%date% %time%

4.5 creation of scheduled task plan

1. Open compmgmt. As follows msc
   
2. New scheduled task execution plan
   
3. General settings
   
   

1. Trigger settings
   
   

2. Operation settings
   Contents filled in the program or script box: D: \ Hehuan_ test\rsync_ workspace\rsync. bat >> D:\hehuan_ test\rsync_ workspace\rsync. log
   

3. Scheduled execution plan setting completed
   

Are you trying to perform Rsync from Linux to Windows share?

This guide is for you.

Rsync is a command-line utility that helps us to copy files from one location to another. We did experiments and found that it is fast. It can also be used for mirroring data and incremental backups.

Here at Ibmi Media, as part of our Server Management Services, we regularly help our Customers to perform Rsync related queries.

In this context, you shall learn how to transfer Rsync from Linux to Windows share.

More information about Rsync ?

A fast and versatile command-line utility, Rsync (Remote Sync) helps to synchronize files and directories between two locations over a remote shell, or from/to a remote Rsync daemon.

In addition, it provides fast incremental file transfer by transferring only the differences between the source and the destination.

We can use Rsync for mirroring data, incremental backups, copying files between systems. It acts as a replacement for scp, sftp, and cp commands.

Benefits and features of the Rsync command include:

i. Rsync efficiently copies and sync files to or from a remote system.

ii. Supports copying links, devices, owners, groups, and permissions.

iii. It’s faster than SCP (Secure Copy) since it allows the transfer of just the differences between two sets of files.

iv. It consumes less bandwidth using the compression and decompression method while sending and receiving data.

How to transfer Rsync from Linux to Windows share ?

To get started, we make a mount point:

mkdir /mnt/share

Then, we mount the smb share:

mount -t cifs -o username=domainusername //ip_add/ShareFolder /mnt/share

It will prompt for a password. It is always a good practice not to provide password within commands.

In addition, if we run the following command we can verify if it has mount:

mount
//ip_add/ShareFolder on /mnt/share type cifs (rw)

Similarly, we can mount it after every boot.

In order to do that we need to add it to /etc/fstab:

//ip_add/ShareFolder /mnt/share smbfs username=domainusername,password=P@S5W0rd 0 0

Now we can Rsync data to it.

For example, to rsync ibmimedia’s home directory into a dir called ‘homedir’ we run:

rsync -avz /home/ibmimedia/ /mnt/share/homedir/

[Stuck with the transfer with rsync? We are here to help you. ]

This procedure may be helpful for backups of Linux VMs on the cloud. Some (most) service providers make it notoriously difficult to export VM images or snapshots out of the cloud.

While it is still possible to perform an image backup using Clonezilla, again, not all service providers allow boot on an ISO for a given VM. In these cases, a helper volume should be added to the VM and have Clonezilla installed on it, and selected as the boot device. The original boot volume would then be backed-up to an image locally on the helper volume and exported out of the cloud after the process completes. Although that is the theory, I have not tested this specific process myself. Keep in mind that some providers VM instances only support one volume attached. Also, this process is a one-shot operation, and is hardly automated, and creates downtime.

There are other block level device time consistent backup strategies you can try though if you use LVM and have space on the LVM group, it is possible to snapshot your volume, mount the snapshot, and have rsync transfer it over the network. Keep in mind that if the rsync process is initiated on the backup system rather than on the backed-up system, you’ll better have to use ssh or other remote command tools to perform snapshot creation and mount and then rsync, have a way to check for errors returned by these commands and then only initate rsync on the backup system. Then perform a post-backup remote execution to unmount the snapshot and destroy the snapshot. This whole process is a bit similar to the Windows VSS snapshot operation.

For more information about this process, check :

LVM snapshot Backup and restore on linux

If you don’t use LVM then you’ll have to resort to backups that are not time consistent (not frozen in time), but it is better than nothing. That is the object of this tutorial.

Setting up cygwin on Windows.

Download cygsetup and perform a full or a minimal install. For a minimal install, you’ll need SSH and Rsync as well as it’s dependencies.

Now you’ll have to choose between using rsync through a SSH encrypted channel, or through a rsync daemon running on the remote host to be backed up.

Using rsync through SSH for a full system backup requires a root login through SSH, and automating rsync with SSH will require a way to supply credentials automatically, unless using a public/private key non interactive (passphrase less) authentication scheme.

In the case of SSH plain password authentication, supplying it to rsync can be done through sshpass, which exists as a Cygwin package, but I have not tested it myself in conjunction with rsync

http://www.cygwin.com/packages/summary/sshpass.html

https://stackoverflow.com/questions/3299951/how-to-pass-password-automatically-for-rsync-ssh-command

However, allowing SSH password root authentication plus storing its password as cleartext in a file for sshpass to use it is a huge security risk.

At least, with a passphrase less public/private key pair, the problem amounts to securing the private key well on the filesystem. It will still be acessible (read only) in the user account context that runs the rsync/ssh script on Windows.

For all these reasons, I find it preferable to use the rsync daemon on the remote system. Which allows to use a backup account instead of root.

The downsides however are that you need to open the rsync TCP port on the remote system and configure your firewalls in the connection path accordingly; and also rsync daemon does not encrypt traffic. If it is an issue, then use a VPN or an IpSec tunnel.

Setting up the rsync daemon on linux Debian

apt-get install rsync

Edit the rsync daemon configuration file.

vi /etc/rsyncd.conf

Here is a sample of the rsyncd.conf

uid=root
gid=root

[share_system_backup]
	path = /
	comment = system root
	read only = true
	auth users = backup
	secrets file = /etc/rsyncd.secrets
	hosts allow = <ip_of_backup_system>

As per the rsync manpage, rsyncd runs as root, and uid and gid parameters, which can be global to all rsync shares or share specific, specify the impersonated context of the rsync access to the filesystem.

Since we’ll perform a system wide backup, we’ll use root.

auth users specify the rsync users authorized to access the share. These users are rsync specific, not system users.

read only is used since no writes will be performed on the backed up system, unless you want rsync to upload some state file on the remote system after backup, as we won’t be using SSH, that is a trick way to signal something to the backed up system, without any remote execution.

hosts_allow is useful for a cloud VM that does not have firewalling options provided by the service provider.

The user login password pair is specified in /etc/rsyncd.secrets.

vi /etc/rsyncd.secrets

backup:346qsfsgSAzet

Use a strong password.

Next start the rsync daemon, check it runs and check its socket is listening on TCP port 873.

rsync --daemon

ps-auwx | grep rsync && netstat -nlp | grep rsync

Then we’ll make rsync launch at system boot.

vi /etc/default/rsync

Change RSYNC_ENABLE to true to start the daemon at system startup through init.d

Rsync configuration on Windows

We’ll start by setting up the windows batch files that will prepare the cygwin environment and run rsync. Change <IP> with your remote system to be backed up IP or DNS name. This script assumes standard port 873

The default file is named CWRSYNC.CMD and should reside at the root of the cygwin64 base folder.

@ECHO OFF
REM *****************************************************************
REM
REM CWRSYNC.CMD - Batch file template to start your rsync command (s).
REM
REM *****************************************************************

REM Make environment variable changes local to this batch file
SETLOCAL

REM Specify where to find rsync and related files
REM Default value is the directory of this batch file
SET CWRSYNCHOME=%~dp0

REM Create a home directory for .ssh 
IF NOT EXIST %CWRSYNCHOME%\home\%USERNAME%\.ssh MKDIR %CWRSYNCHOME%\home\%USERNAME%\.ssh

REM Make cwRsync home as a part of system PATH to find required DLLs
SET CWOLDPATH=%PATH%
SET PATH=%CWRSYNCHOME%\bin;%PATH%

REM Windows paths may contain a colon (:) as a part of drive designation and 
REM backslashes (example c:\, g:\). However, in rsync syntax, a colon in a 
REM path means searching for a remote host. Solution: use absolute path 'a la unix', 
REM replace backslashes (\) with slashes (/) and put -/cygdrive/- in front of the 
REM drive letter:
REM 
REM Example : C:\WORK\* --> /cygdrive/c/work/*
REM 
REM Example 1 - rsync recursively to a unix server with an openssh server :
REM
REM       rsync -r /cygdrive/c/work/ remotehost:/home/user/work/
REM
REM Example 2 - Local rsync recursively 
REM
REM       rsync -r /cygdrive/c/work/ /cygdrive/d/work/doc/
REM
REM Example 3 - rsync to an rsync server recursively :
REM    (Double colons?? YES!!)
REM
REM       rsync -r /cygdrive/c/doc/ remotehost::module/doc
REM
REM Rsync is a very powerful tool. Please look at documentation for other options. 
REM

REM ** CUSTOMIZE ** Enter your rsync command(s) here

echo "start" >> c:\scripts\rsync_sys.log
date /t >> c:\scripts\rsync_sys.log
time /t >> c:\scripts\rsync_sys.log

rsync --no-perms --itemize-changes --password-file=rsync_p -lrvcD --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/root/*"} backup@<IP>::share_system_backup/ /cygdrive/d/BACKUPS_FROM_CLOUD/SYSTEM >> c:\scripts\rsync_sys.log

date /t >> c:\scripts\rsync_sys.log
time /t >> c:\scripts\rsync_sys.log
echo "stop" >> c:\scripts\rsync_sys.log

You’ll need to add a file named rsync_p in my example that contains the password specified for the backup user, matching the password defined on the rsync daemon host. This allows non interactive execution of rsync.

Place this file at the cygwin64 base folder level.

Secure that file well at the NTFS level.

itemize-changes will log in the rsync_sys.log the file operations performed on the system. It is useful for troubleshooting.

http://www.staroceans.org/e-book/understanding-the-output-of-rsync-itemize-changes.html

The -lrvc flags tells rsync to copy symbolic link information (absolute and relative). You can test that is the case by using ls -ltra through a cygwin shell, recurse directories, be verbose, and check for changes based on checksum instead of file timestamps, which could be useful if there are timestamp discrepancies because of the cygwin environment. It is worth testing it’s effect with a test run.

Also, I excluded the standard list of directories recommended to be excluded from a system backup for Debian systems using the –exclude option.

Permissions / ACL issues

Note that I use –no-perms for rsync. That is one caveat with rsync running on cygwin, since it uses POSIX to Windows ACL translation, copying perms COULD render the files unreadable by other processes on the Windows system if permissions from the source system are preserved, On the other hand, preserving source permissions could make baremetal recovery possible. It is worth experimenting with this option.

A way to circumvent this problem when using –no-perms is to back up permissions separately on the linux system using getfacl and setfacl for restore operations. keep in mind that getfacl ouput is substantial in terms of size and getfacl does not support directory exclusions. It may involve a bit of scripting.

Example of getfacl script (backupacl.sh)

cd /home/backup
getfacl -RL /bin > filesystem_acls_bin.dat
getfacl -RL /boot > filesystem_acls_boot.dat
getfacl -RL /etc > filesystem_acls_etc.dat
getfacl -RL /home > filesystem_acls_home.dat
getfacl -RL /lib > filesystem_acls_lib.dat
getfacl -RL /lib64 > filesystem_acls_lib64.dat
getfacl -RL /media > filesystem_acls_media.dat
getfacl -RL /mnt > filesystem_acls_mnt.dat
getfacl -RL /opt > filesystem_acls_opt.dat
getfacl -L /root > filesystem_acls_root.dat
getfacl -RL /sbin > filesystem_acls_sbin.dat
getfacl -RL /snap > filesystem_acls_snap.dat
getfacl -RL /srv > filesystem_acls_srv.dat
getfacl -RL /tmp > filesystem_acls_tmp.dat
getfacl -RL /usr > filesystem_acls_usr.dat
getfacl -RL /var > filesystem_acls_var.dat

tar -czvf backup_acls.tar.gz filesystem_acls_*
rm -f filesystem_acls_*

You can use xargs and a $ placeholder plus a list of folders to backup permissions from to make this script a one-liner.

the backup_acls.tar.gz will be backed up by rsync when it runs.

In this example, ACL backups and the backup script are stored in /home/backup. As said before, backup is not necessarily a linux user since rsync maintains its own authentication database, but I use this directory for convenience.

In this script backup of /root ACLs is not recursive as a dirty way to remove the .wine directory that contains symbolic links to the whole file system, and thus would make getfacl to follow them and spend a huge useless time (and filespace) for directories such as /dev, /proc /sys. So it’s possible for you to add the recurse flag if this problem does not affect your system.

Run the batch for tests and schedule it by adding it to /etc/cron.daily/ for instance. It should run in the root context.

A note about restores and symbolic links

the rsync -l options copies symbolic links, which means thar you have to use rsync too for restore operations, Don’t expect symbolic links to magically reappear if you use WinSCP or another tool to restore a folder.

More on this subject :

https://www.baeldung.com/linux/rsync-copy-symlinks

Testing the backup.

Edit the cwrsync.cmd to add the dry-run option, run the cwrsync.cmd through a cmd shell and examine the rsync_sys.log file, If everything seems OK, then you can remove the dry-run option, and run it again. The first run will take a long time, use the start and stop markers in the logfile to check how long it takes once finished. rsync will tell you the speedup due to the delta algorithm, it should increase on subsequent runs.

Scheduling the backup.

Use Windows task scheduler to run the cwrsync.cmd script. In my case, I use a privileged account. It is advised to test with the least amount of privileges for cygwin to be happy and for filesystem operations to succeed.

It is advised the turn on scheduled tasks history. And very important, make sure that the “start in” directory for the cwrsync.cmd action is set to the base cygwin64 folder.

Test the backup run using a “run once” trigger at t+1 minute in addition to the recurrent trigger.

A final note.

This backup strategy will ensure that you have an updated state of your remote system on the Windows system. It is similar to an incremental backup that patches the destination directory. It does not store any history of files besides the last backup operation.

I don’t use the –delete option that propagates deletions of files and folders from the source to the destination, has it can be very impactful for a disaster recovery use where there is no file versioning history. Note however, that this will bloat the backed up structure on the destination and make recovery efforts more complicated in the long term.

The best option for a backup if you use LVM, albeit using more space on the destination is to use cygwin + rsync to backup a tar gz file of a LVM snapshot directory structure instead of the directory tree of “/”. If you go this way though, it is advised, as already stated above, to perform the snapshot operations synchronously from the backup system through the cwrsync.cmd using SSH, before the rsync operations. That probably means using a passphrase less public/private key pair for ssh to run these commands interactively. Don’t forget to unmount and remove the snapshot after rsync is done.