What is the difference between merge --squash and rebase?

Both git merge --squash and git rebase --interactive can produce a "squashed" commit.
But they serve different purposes.

• git merge --squash abranch

will produce a squashed commit on the destination branch, without marking any merge relationship.
(Note: it does not produce a commit right away: you need an additional git commit -m "squash branch")
This is useful if you want to throw away the source branch completely, going from (schema taken from SO question):

 git checkout stable      X                   stable     /                   a---b---c---d---e---f---g tmp

to:

git merge --squash tmpgit commit -m "squash tmp"      X-------------------G stable     /                   a---b---c---d---e---f---g tmp

and then deleting tmp branch.

Note: git merge has a --commit option, but it cannot be used with --squash. It was never possible to use --commit and --squash together.
Since Git 2.22.1 (Q3 2019), this incompatibility is made explicit:

See commit 1d14d0c (24 May 2019) by Vishal Verma (reloadbrain).
^{(Merged by Junio C Hamano -- gitster -- in commit 33f2790, 25 Jul 2019)}

merge: refuse --commit with --squash

Previously, when --squash was supplied, 'option_commit' was silently dropped. This could have been surprising to a user who tried to override the no-commit behavior of squash using --commit explicitly.

git/git builtin/merge.c#cmd_merge() now includes:

if (option_commit > 0)    die(_("You cannot combine --squash with --commit."));

• git rebase --interactive

replays some or all of your commits on a new base, allowing you to squash (or more recently "fix up", see this SO question), going directly to:

git checkout tmpgit rebase -i stable      stable      X-------------------G tmp     /                     a---b

If you choose to squash all commits of tmp (but, contrary to merge --squash, you can choose to replay some, and squashing others).

So the differences are:

• squash does not touch your source branch (tmp here) and creates a single commit where you want.
• rebase allows you to go on on the same source branch (still tmp) with:
  • a new base
  • a cleaner history

Merge commits: retains all of the commits in your branch and interleaves them with commits on the base branch

Merge Squash: retains the changes but omits the individual commits from history

Rebase: This moves the entire feature branch to begin on the tip of the master branch, effectively incorporating all of the new commits in master

More on here

Merge squash merges a tree (a sequence of commits) into a single commit. That is, it squashes all changes made in n commits into a single commit.

Rebasing is re-basing, that is, choosing a new base (parent commit) for a tree. Maybe the mercurial term for this is more clear: they call it transplant because it's just that: picking a new ground (parent commit, root) for a tree.

When doing an interactive rebase, you're given the option to either squash, pick, edit or skip the commits you are going to rebase.

Hope that was clear!