xsync

A simple, self-contained rsync replacement that travels over SSH and uses CRC32 to determine which files need transferring. No dependencies on the remote host beyond SSH and the xsync binary itself; which it uploads automatically on first use.

Features

• Single binary — copy xsync to every machine, or let it deploy itself
• CRC32 diffing — only transfers files whose checksum differs from the remote
• Chunked streaming — files sent in 256 KiB chunks, no RAM limit on file size
• End-to-end verification — CRC32 confirmed by the remote after every write
• Symlink support — symlinks are preserved as symlinks, not followed
• Deletion mode — -d mirrors the source, removing remote-only files
• Parallel destinations — sync to multiple hosts simultaneously
• Pipelined transfers — next file streams while waiting for the previous ACK
• Live progress bar — per-destination progress with bytes and filename (TTY-aware)
• Self-deploying — uploads itself to /tmp/.xsync_agent on the remote if missing or stale
• SSH config aware — reads ~/.ssh/config (or --ssh-dir) for HostName, User, Port, IdentityFile
• ed25519 preferred — always negotiates ssh-ed25519 host keys first
• Accept-new host keys — unknown hosts are added to known_hosts automatically; changed keys are hard-rejected
• Custom SSH directory — --ssh-dir supports XDG, per-identity, and non-standard SSH layouts

License

Usage

xsync [options] <source> <user@host:dest> [user@host2:dest2 ...]

  source            Local file or directory to sync
  user@host:dest    One or more remote destinations

Options:
  -i <keyfile>      SSH identity file (default: auto-detected)
  --ssh-dir <dir>   SSH config directory (default: auto-detected)
                    Checked in order: $XDG_CONFIG_HOME/ssh, ~/.config/ssh,
                    ~/.local/ssh, ~/.ssh
  -p <port>         SSH port (default: 22, or from ssh_config)
  -r                Recursive — sync directory and all subdirectories
  -d                Delete remote files not present in source (mirror mode)
  -v                Verbose output
  -n                Dry run — show what would be transferred, send nothing
  --version         Print version and exit

Examples

# Sync a single file to one host
xsync myapp user@web1.example.com:/usr/local/bin/myapp

# Sync a directory to three hosts at once
xsync -r ./dist/ user@web1:/var/www/ user@web2:/var/www/ user@web3:/var/www/

# Mirror a directory (delete remote files that no longer exist locally)
xsync -r -d ./dist/ user@web1:/var/www/

# Dry run to see what would change
xsync -n -r ./config/ deploy@db1.local:/etc/myapp/

# Use a specific SSH key and non-standard port
xsync -i ~/.ssh/deploy_key -p 2222 -r ./app/ prod@10.0.0.5:/opt/app/

# Use a non-standard SSH directory (XDG, per-identity, etc.)
xsync --ssh-dir ~/.config/ssh -r ./z-1/ backup.local:/home/moo/z-1/

# Host alias resolved via ssh_config — no extra flags needed
# (assuming ~/.config/ssh/config has a Host entry for "backup.local")
xsync -r ./videos/ backup.local:/home/moo/videos/

How it works

Sender                                    Remote Agent
  |                                           |
  |-- SSH connect (Go library) ------------>  |
  |-- upload xsync binary (if stale) ------>  |
  |                                           |
  |-- spawn: ssh user@host xsync --agent -->  |
  |                                           |
  |-- MSG_MANIFEST_REQ (destPath) --------->  |
  |<- MSG_MANIFEST_RESP ([]{path,crc32}) --   |
  |                                           |
  |  [diff: local CRC32 vs remote manifest]   |
  |                                           |
  |-- MSG_FILE    (FileHeader) ------------>  |  ─┐
  |-- MSG_CHUNK   (≤256 KiB) -------------->  |   │ per changed file
  |-- MSG_CHUNK   (≤256 KiB) -------------->  |   │ (pipelined with
  |-- MSG_CHUNK_END (empty) --------------->  |   │  next file send)
  |<- MSG_ACK    (verified CRC32) ---------   |  ─┘
  |                                           |
  |-- MSG_SYMLINK (path, target) ---------->  |  per symlink
  |<- MSG_ACK -----------------------------   |
  |                                           |
  |-- MSG_DELETE (path) ------------------->  |  if -d flag
  |<- MSG_ACK -----------------------------   |
  |                                           |
  |-- MSG_DONE ---------------------------->  |
  |<- MSG_DONE ----------------------------   |

1. xsync SSHes into each target host using the Go SSH library
2. It checks /tmp/.xsync_agent on the remote — if missing or a different size, it uploads itself via cat
3. It spawns xsync --agent on the remote using the system ssh binary (so ProxyJump, ControlMaster, etc. all work), passing -F <sshdir>/config explicitly
4. The agent walks the destination with os.Lstat and returns a manifest of every file and symlink with CRC32 checksums
5. The sender diffs local CRC32+size against the manifest — unchanged files are skipped
6. Changed files are streamed in 256 KiB chunks; the agent writes each chunk directly to a temp file, hashing as it goes, then atomically renames it into place
7. The agent verifies the final CRC32 and sends an ACK — the sender confirms it matches
8. Symlinks are sent as MSG_SYMLINK (target string only, no data chunks)
9. If -d is set, remote entries absent from the local source are deleted after all transfers
10. All destinations run in parallel goroutines with independent progress bars

SSH config support

xsync reads your SSH config file before connecting. A Host entry is all you need:

# ~/.config/ssh/config
Host backup.local
    HostName 192.168.0.100
    User moo
    Port 22
    IdentityFile ~/.config/ssh/gpg/alisteratkamikishi/id_ed25519

Then just run:

xsync --ssh-dir ~/.config/ssh -r ./dist/ backup.local:/home/moo/dist/

HostName, User, Port, and IdentityFile are all applied automatically. CLI flags (-i, -p, explicit user@host) always override the config file.

SSH directory auto-detection

When --ssh-dir is not specified, xsync searches for your SSH directory in order:

1. $XDG_CONFIG_HOME/ssh
2. ~/.config/ssh — if it contains known_hosts, config, or any id_* file
3. ~/.local/ssh — same check
4. ~/.ssh — unconditional fallback

Building

# Current platform
go build -ldflags="-s -w" -o xsync .

# Cross-compile
GOOS=linux  GOARCH=amd64  go build -ldflags="-s -w" -o xsync-linux-amd64  .
GOOS=linux  GOARCH=arm64  go build -ldflags="-s -w" -o xsync-linux-arm64  .
GOOS=darwin GOARCH=arm64  go build -ldflags="-s -w" -o xsync-darwin-arm64 .
GOOS=darwin GOARCH=amd64  go build -ldflags="-s -w" -o xsync-darwin-amd64 .

# All platforms via Makefile
make all-platforms

Requires golang.org/x/crypto (the only external dependency).

Source layout

File	Purpose
main.go	Argument parsing, mode dispatch
proto.go	Binary framing protocol (length-prefixed messages, message types)
agent.go	Remote agent — manifest, chunked receive, atomic write, symlinks, deletion
sender.go	Sender — file collection, CRC32 diff, chunked stream, pipelined ACKs
sshconn.go	SSH connection, host key management, binary self-upload
sshconfig.go	ssh_config parser — HostName, User, Port, IdentityFile, glob matching
progress.go	TTY-aware per-destination progress bar
Makefile	Cross-compile targets

Known limitations

• No recursive directory deletion under -d (individual files and symlinks only)
• No bandwidth throttling (-bwlimit)
• No resume support for interrupted transfers (restarts from zero)
• No delta compression (whole files only; no rolling-checksum patching)
• Windows sender untested (--agent mode works on any POSIX remote)