Removing duplicates on a variable without sorting Removing duplicates on a variable without sorting unix unix

Removing duplicates on a variable without sorting


new_variable=$( awk 'BEGIN{RS=ORS=" "}!a[$0]++' <<<$variable );

Here's how it works:

RS (Input Record Separator) is set to a white space so that it treats each fruit in $variable as a record instead of a field. The non-sorting unique magic happens with !a[$0]++. Since awk supports associative arrays, it uses the current record ($0) as the key to the array a[]. If that key has not been seen before, a[$0] evaluates to '0' (awk's default value for unset indices) which is then negated to return TRUE. I then exploit the fact that awk will default to 'print $0' if an expression returns TRUE and no '{ commands }' are given. Finally, a[$0] is then incremented such that this key can no longer return TRUE and thus repeat values are never printed. ORS (Output Record Separator) is set to a space as well to mimic the input format.

A less terse version of this command which produces the same output would be the following:

awk 'BEGIN{RS=ORS=" "}{ if (a[$0] == 0){ a[$0] += 1; print $0}}'

Gotta love awk =)

EDIT

If you needed to do this in pure Bash 2.1+, I would suggest this:

#!/bin/bash    variable="apple lemon papaya avocado lemon grapes papaya apple avocado mango banana"temp="$variable"new_variable="${temp%% *}"while [[ "$temp" != ${new_variable##* } ]]; do   temp=${temp//${temp%% *} /}   new_variable="$new_variable ${temp%% *}"doneecho $new_variable;


This pipeline version works by preserving the original order:

variable=$(echo "$variable" | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)


Pure Bash:

variable="apple lemon papaya avocado lemon grapes papaya apple avocado mango banana"declare new_value=''for item in $variable; do  if [[ ! $new_value =~ $item ]] ; then   # first time?    new_value="$new_value $item"  fidonenew_value=${new_value:1}                  # remove leading blank