how to remove dupliace lines in authorized_keys files on remote machines

Question

we have sometimes huge file of /root/.ssh/authorized_keys on our linux machines , and that because a lot of duplicate lines in the file as the following

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6yaJuzX2QldXj9jI/IYbJQuYDTUf232IbkefUDG4sZxkkScbiqC4skJs9bC58iovYxMVLB7YijIHDri7ONfKzixooSfpf8x18JdmSTkEl7WVTPm3TI/fPVP7DDOoBbqpTeZzS6cFVRMceve3ecFp/Z
D02RfLy6FHu3Y9o55g4Hlm+IgRq+QflsSoY3khZhaxofyzYIchg9NI1RzEZJQEBIMlQZMd+bRiBoAtzqI2BtKd5YmnBmxGHhnZLswSeo7hz+2cAPe+Ng37V91cSuygQJyKf20f1DmhSKHvHEDU3EXDPbjO8H0LNz6OEhsjwUj+G5dcJA04wY0Y1+qCfRz
kR [email protected]
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6yaJuzX2QldXj9jI/IYbJQuYDTUf232IbkefUDG4sZxkkScbiqC4skJs9bC58iovYxMVLB7YijIHDri7ONfKzixooSfpf8x18JdmSTkEl7WVTPm3TI/fPVP7DDOoBbqpTeZzS6cFVRMceve3ecFp/Z
D02RfLy6FHu3Y9o55g4Hlm+IgRq+QflsSoY3khZhaxofyzYIchg9NI1RzEZJQEBIMlQZMd+bRiBoAtzqI2BtKd5YmnBmxGHhnZLswSeo7hz+2cAPe+Ng37V91cSuygQJyKf20f1DmhSKHvHEDU3EXDPbjO8H0LNz6OEhsjwUj+G5dcJA04wY0Y1+qCfRz
kR [email protected]
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6yaJuzX2QldXj9jI/IYbJQuYDTUf232IbkefUDG4sZxkkScbiqC4skJs9bC58iovYxMVLB7YijIHDri7ONfKzixooSfpf8x18JdmSTkEl7WVTPm3TI/fPVP7DDOoBbqpTeZzS6cFVRMceve3ecFp/Z
D02RfLy6FHu3Y9o55g4Hlm+IgRq+QflsSoY3khZhaxofyzYIchg9NI1RzEZJQEBIMlQZMd+bRiBoAtzqI2BtKd5YmnBmxGHhnZLswSeo7hz+2cAPe+Ng37V91cSuygQJyKf20f1DmhSKHvHEDU3EXDPbjO8H0LNz6OEhsjwUj+G5dcJA04wY0Y1+qCfRz
kR [email protected]

what is the best approach to remove this duplicate lines so only unique lines will be appears?

we need to remove the duplicate lines VIA ssh on remote machines

Answer 1

Commands can be run on the remote server with the ssh client

ssh hostname '
    cd /root/.ssh; 
    cp -a authorized_keys authorized_keys.orig;
    sort -u authorized_keys -o authorized_keys
'

This is written as multi-line code for readability. The whole thing can also be on one line (in which case there is no need to enclose it in '', in this case).

If a backup (.orig) is absolutely certainly not needed just remove the cp ... line.

Doing it this way clearly changes the order of lines. If that is a problem you can run a script, or a one-liner, that can preserve the order.

---

If there is more to be done, one way to run a one-liner on the remote server is

ssh hostname << 'CMD'
cd /root/.ssh/
perl -i.orig -wne'$uniq{$_} = 1; }{ print for keys %uniq' authorized_keys
CMD

The }{ syntax starts an END block, which runs after all lines have been processed. If a backup is certainly not needed remove .orig, so leave only -i switch (to change the file in-place).

The above still merely removes duplicates but one can substitute or amend the code under '' with other Perl code that may be needed. For example, to preserve order among unique lines can use this Perl command-line program ("one-liner") instead of the one above

perl -MList::Util=uniq -i.orig -wne'
    push @lines, $_; END { print for uniq @lines }' authorized_keys

Here we use uniq from the core module List::Util, which returns unique elements from its input list. It keeps the first of (possibly) repeated elements and maintains the order. I assume that the authorized_keys file cannot be too large to read all into memory.

See "Command switches" in perlrun for more about one-liners.

Note that the keyword in the "heredoc" syntax used above is under quotes ('CMD'), to suppress variable expansion by the shell.

The examples relating to possibly more elaborate needs here use Perl as it was among the languages originally tagged in the question but of course any other language can be used.