Linux utilities

Secure Connection

  • One of the most used tool types by Linux administrators are remote tools
  • Telnet has long been the most used one
  • However, telnet is vulnerable for network traffic monitoring since all information is sent in plain text format without any encryption
  • Tatu Ylönen developed the solution for this problem, nowadays known as SSH (Secure Shell)

SSH (Secure Shell)

  • SSH provides encrypted connection between two computers
  • SSH is not installed in newer Ubuntu distributions by default and can be installed with the following command:
testuser@ubuntu-PC:~$ sudo apt install ssh
  • To check whether the ssh daemon has started run the following command:
testuser@ubuntu-PC:~$ sudo systemctl status ssh
[sudo] password for testuser: 
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-10-08 14:05:10 EEST; 20min ago
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 5641 (sshd)
      Tasks: 1 (limit: 4657)
     Memory: 1.2M
     CGroup: /system.slice/ssh.service
             └─5641 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

loka 08 14:05:10 ubuntu-PC systemd[1]: Starting OpenBSD Secure Shell server...
loka 08 14:05:10 ubuntu-PC sshd[5641]: Server listening on 0.0.0.0 port 22.
loka 08 14:05:10 ubuntu-PC sshd[5641]: Server listening on :: port 22.
loka 08 14:05:10 ubuntu-PC systemd[1]: Started OpenBSD Secure Shell server.
  • Like shown from the example above ssh service is running and accepting remote connections to port 22
  • Now clients can connect to this computer from for example another Ubuntu computer using either of the following commands:
  • Syntax 1
ssh target_computer [-p port_number]
  • Syntax 2
ssh user@target_computer [-p port_number]
  • Target computer can be identified by its IP address or hostname
  • Example: Open SSH connection for the remote host palvelin.pilvipalvelut.fi
testuser@ubuntu-PC:~$ ssh testuser@palvelin.pilvipalvelut.fi
The authenticity of host ‘palvelin.pilvipalvelut.fi (195.168.20.200)' can't be established.
RSA key fingerprint is 44:de:ce:f6:10:ba:be:fa:2b:d1:35:94:45:9d:74:ba.
Are you sure you want to continue connecting (yes/no)? yes

Using Putty as SSH client

  • Putty is the client program for SSH and telnet connections
  • SSH connection can be formed for local or remote computer running ssh service
  • Putty can be downloaded for Windows computers from here
  • In order to connect to your Ubuntu installation running inside VirtualBox, follow this guide:
    1. First you need to make a port forward for your Ubuntu since it is running behind NAT network by default
      • Select your virtual machine settings
      • Then under Network tab select Adapter 1 and open Advanced menu
      • Select Port Forwarding
      • Click the icon Adds new port forwarding rule.
      • Fill the following information:
        • Name: SSH
        • Protocol: TCP
        • Host IP: 127.0.0.1
        • Host Port: 22
        • Guest IP: leave empty
        • Guest Port: 22
      • Save changes and exit settings menu by selecting OK to all previous open windows
    2. Start your Ubuntu from VirtualBox
      • If you haven't installed SSH yet, follow the installation guidance presented in previous section
    3. Open Putty
      • Fill the following information:
        • Host Name (or IP address): 127.0.0.1
        • Port: 22
        • Connection type:: SSH
      • Click Open
      • Connection should now be established

SCP (Secure Copy)

  • SSH connection can also be used for copying files and directories between computers
  • For this kind of operation scp command is used
  • The syntax can be found from the figure below

SCP command syntax

  • Below are examples of using scp command

  • Example 1: Copy sensors.txt file from local computer to remote computer

testuser@ubuntu-PC:~$ ls
sensors.txt
testuser@ubuntu-PC:~$ scp sensors.txt testuser@palvelin.pilvipalvelut.fi:
testuser@palvelin.pilvipalvelut.fi's password:
sensors.txt                                                 100%  367     0.4KB/s   00:00
  • Example 2: Copy sensors.txt file from remote computer to local computer and give it a new name sensors_new.txt
testuser@ubuntu-PC:~$ ls
sensors.txt
testuser@ubuntu-PC:~$ scp testuser@palvelin.pilvipalvelut.fi:sensors.txt sensors_new.txt
testuser@palvelin.pilvipalvelut.fi's password:
sensors.txt                                                 100%  103KB 102.8KB/s   00:00
testuser@ubuntu-PC:~$ ls
sensors.txt sensors_new.txt

WGET

  • wget is a command for downloading files from internet
  • WGET supports HTTP, HTTPS and FTP protocols
  • WGET is non-interactive meaning it can also be running in the background
  • Syntax for wget command: wget options url
  • Example: Download the installation package for GIMP program
testuser@ubuntu-PC:~$ wget https://download.gimp.org/pub/gimp/v2.10/gimp-2.10.12.tar.bz2
--2019-02-22 07:42:24-- https://download.gimp.org/pub/gimp/v2.10/gimp-2.10.12.tar.bz2
Resolving download.gimp.org (download.gimp.org)... 209.132.180.179
Connecting to download.gimp.org (download.gimp.org)|209.132.180.179|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32614932 (31M) [application/x-bzip2]
Saving to: ’gimp-2.10.12.tar.bz2’
gimp-2.10.12.tar.bz2       43%[=============>                    ]  13.54M   503KB/s   eta 25s

CURL

  • curl is an alternative program for downloading files from internet, but it also has many other features that are not possible with wget command
  • CURL supports many protocols:
    • DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP
  • In addition to downloading files from internet curl can also be used to transfer files to remote computer like with scp command or for example list the directory content from remote FTP server
  • Example: Copy file sensor.txt to remote server server.pilvipalvelut.fi
testuser@ubuntu-PC:~$ curl -F ’sensor_file=@/home/testuser/sensor.txt’ https://server.pilvipalvelut.fi

TAR (Tar Archiving Utility)

  • TAR is used for adding directories into one archive file
  • All permissions for files and directories will be preserved within the archive
  • TAR does not include compression feature (file size will not be decreased)

  • Examples of using tar command

  • Example 1: Add data directory into one tar archive

testuser@ubuntu-PC:~$ tar cf data_backup.tar data/*
  • Example 2: Extract tar archive content to directory
testuser@ubuntu-PC:~$ tar xf data_backup.tar
  • Example 3: Add new_sensors directory to the end of data_backup.tar archive
testuser@ubuntu-PC:~$ tar rf data_backup.tar new_sensors/
  • Example 4: Print the content of data_backup.tar archive
testuser@ubuntu-PC:~$ tar tf data_backup.tar

GZIP (Compress or Expand Files)

  • Used for compressing a file (file size decreases)
  • Important: gzip program transforms file into .gz format and original file will be deleted!

  • Examples of using gzip command

  • Example 1: Compress sensors.txt file into .gz format

testuser@ubuntu-PC:~$ gzip sensors.txt
  • Example 2: Remove gzip compression from sensors.txt file
testuser@ubuntu-PC:~$ gzip –d sensors.txt.gz

Linux text editors

  • Text editors are useful in Linux systems and are especially used in editing system and program configuration files
  • There are dozens of different text editors available for command line and graphical user interface
  • Each Linux distribution with graphical user interface has at least one graphical text editor and one command line based text editor
  • Important: in server distributions there are usually no graphical user interface available and thus learning to use at least one CLI based text editor is essential!
  • Some common text editors for GUI and CLI are listed below
  • GUI editors have also the desktop environment mentioned where this particular editor can be found by default
GUI editors
Gedit (GNOME)
Emacs (GNU)
Leafpad (LXDE)
Kate/KWrite (KDE)
Mousepad (Xfce)
CLI editors
Pico
Nano
Vi(m)
Emacs

Nano

  • Nano is a simple and easy-to-use text editor for CLI environments
  • Nano is translated in finnish and it supports UTF-8 encoding and source code coloring
  • Nano can be started in two ways:
  • Method 1: Open empty file / buffer
testuser@ubuntu-PC:~$ nano
  • Method 2: Open nano by providing new file name as a parameter. If file does not exist, it will be created. Otherwise the existing file will be opened in Nano
testuser@ubuntu-PC:~$ nano sensor-settings.cfg
  • Inside Nano editor cursor can be moved with arrow keys
  • List of possible hot keys is presented at the bottom of the screen (^ character means Ctrl button)
  • Editor can be closed and file saved with the following list of actions:
    1. Ctrl + x Start closing the file
    2. Y Accept and save changes made for the file
    3. Accept or optionally change the default save name for the file

Nano editor view

  • Below is the list of useful Nano editor key combinations for different actions
Key combination Action
Ctrl + c Tells the cursor location in the text file
Ctrl + r Copy the content from target file into the position of the cursor
Ctrl + k Cut
Ctrl + u Paste
Ctrl + w Search string from the text file
Ctrl + a Move cursor to the beginning of the line
Ctrl + e Move cursor to the end of the line

Vim

  • Vim (Vi improved) is text editor, which have been developed from Vi editor
  • In newer Linux distribution command vi is symbolic link pointing to Vim editor
  • Vim editor is very different compared to Nano editor
  • In practice Vim editor has operating modes which are used to work with the editor (Mode descriptions picked from Vim editor help):
Mode Mode description
Normal mode In Normal mode you can enter all the normal editor commands. If you start the editor you are in this mode (unless you have set the 'insertmode' option, see below). This is also known as command mode.
Visual mode This is like Normal mode, but the movement commands extend a highlighted area. When a non-movement command is used, it is executed for the highlighted area.
Select mode This looks most like the MS-Windows selection mode. Typing a printable character deletes the selection and starts Insert mode.
Insert mode In Insert mode the text you type is inserted into the buffer.
Command-line mode In Command-line mode (also called Cmdline mode) you can enter one line of text at the bottom of the window. This is for the Ex commands, ":", the pattern search commands, "?" and "/", and the filter command, "!".
Ex mode Like Command-line mode, but after entering a command you remain in Ex mode. Very limited editing of the command line.
Terminal-Job mode Interacting with a job in a terminal window. Typed keys go to the job and the job output is displayed in the terminal window.
  • Below is a list of Vim editor basic commands
Command Description
ZZ or :wq or :x Saves the file and exits the editor
:w Saves the file
:w new_file Saves the file with the name new_file
:w! Saves the file without checking writing permissions
:a,bw new_file Saves the content from line a to line b with the name new_file
:a,bw >> new_file Saves the content from line a to line b to the end of file new_file
:q Exits the program without saving
:q! Forces to exit the program without saving
:e Edits the file again without saving changes
:we! Edits the file again and saves the changes before next edit
:u Undo the previous change

Text insertion in Vim

Linux command line viewers

  • Viewers allow user to quickly examine the content of a file without opening it with a text editor
  • The content of a file is printed to the command line
  • Another great use for viewers is to view command output page by page for example
  • The most important viewers are the following:
    • less
    • more
    • cat
  • These are included by default in all modern Linux distributions

Less

  • User may use arrow keys to navigate the command output or file content up and down
  • With less command the whole content is not read to the computer's memory at once
    • This allows the use of less on computers with small amount of memory
    • In addition, the content is shown faster with less command than it would with text editors
  • Example of less command:

Example of less command

  • Very useful feature of less command is the +F option
  • This option allows the real-time monitoring of a file (especially useful for log file monitoring)
  • Example of the usage of +F option:

Second example of less command

  • less can also be used for viewing another commands output like shown in the example below
testuser@ubuntu-PC:~$ ls -la /etc | less
  • With this command you can now navigate directory listing of /etc up and down using arrow keys (quit with q)

More

  • More command is similar to previously presented less command
  • The most notable difference is that with more command output is processed page by page
  • Navigating the output backwards with arrow keys is not possible
  • Example of more command:

Example of more command

  • Similarly to less command, more can be used for examining output of other commands
testuser@ubuntu-PC:~$ ls -la /etc | more

Cat

  • Unlike with less and more command, Cat prints the content of a file to the command line and returns an empty prompt for the user
  • With long files command line window buffer might not be able to view the whole content
  • Cat is usually utilized with pipe when for example a particular word is searched from the output
  • Example of cat command:

Example of cat command

  • Head command prints a requested amount of lines starting from the beginning of the file
  • Line count is given as an option after the command and target file will be presented after the option
  • Example of head command:
    • Print first ten lines from auth.log file

Example of head command

Tail

  • Tail command prints a requested amount of lines from the end of the file
  • Similar to head command line count is given as an option after the command followed by a target file
  • Example of tail command:
    • Print the last four lines from syslog file

Example of tail command

Grep (Print Lines Matching a Pattern)

  • grep is a multipurpose command for searching strings from a file or any given input
  • Grep will return rows from target input or file that match the given search pattern
  • Grep also supports the use of regex (Regular expression), which can be utilized for creating more complicated search patterns
  • The following syntax may be used for grep:
grep string target_file

OR

command | grep string

  • The most important options for grep:

    • -i removes the casesensitivity (small and great letters won't matter)
    • -v prints lines not matching the condition
    • -c prints the amount of matching lines
    • -n prints matching lines with row numbers

  • Below are some examples for using a grep command

  • Example 1: Print rows from process listing where testuser string is included

testuser@ubuntu-PC:~$ psaux | grep testuser
root      1561  0.0  0.1 105684  7044 ?        Ss   09:27   0:00 sshd: testuser[priv]
testuser  1563  0.0  0.1  76640  7060 ?        Ss   09:27   0:00 /lib/systemd/systemd --user
testuser  1564  0.0  0.0 111728  2416 ?        S    09:27   0:00 (sd-pam)
testuser  1653  0.0  0.1 107984  5368 ?        S    09:27   0:00 sshd: testuser@pts/0
testuser  1654  0.0  0.1  21492  5212 pts/0    Ss   09:27   0:00 -bash
root      1731  0.0  0.0  61756  3764 pts/0    S    09:39   0:00 su testuser
testuser  1732  0.0  0.0  20280  3668 pts/0    S    09:39   0:00 bash
testuser  1764  0.0  0.0  38372  3580 pts/0    R+   09:52   0:00 ps aux
testuser  1765  0.0  0.0  13136  1136 pts/0    S+   09:52   0:00 grep testuser
  • Example 2: Find rows from /etc/passwd file with testuser string included in either small or great letters
testuser@ubuntu-PC:~$ grep -i 'testuser' /etc/passwd
testuser:x:1000:1000:testuser:/home/testuser:/bin/bash
  • Example 3: Find rows from /etc/passwd file with UID or GID number including at least one zero (0)
testuser@ubuntu-PC:~$ grep ':00*:' /etc/passwd
root:x:0:0:root:/root:/bin/bash

Find (Search For Files In a Directory Hierarchy)

  • find is a command for finding files from Linux filesystem
  • The search will always be performed for the whole filesystem → search times might be long in installations containing a lot of files
  • find command has the following syntax: find path operators
  • The following parameters are required:
    • Path: The part of the filesystem where search will be performed
    • Operators: Define what should be searched and how search results are handled
  • The most simple example of performing a search with find command will only include the name of the file or directory

  • Below are examples where find command is utilized

  • Example 1: Find files and directories starting with string .bash from testuser's home directory

testuser@ubuntu-pc:~$ find /home/testuser -name ".bash*"
/home/testuser/.bash_logout
/home/testuser/.bash_history
/home/testuser/.bashrc
  • Example 2: Find all files (not directories) starting with string apt from /etc directory
testuser@ubuntu-pc:~$ find /etc -type f -name "apt*"
/etc/kernel/postinst.d/apt-auto-removal
/etc/logrotate.d/apt
find: ‘/etc/ssl/private’: Permission denied
find: ‘/etc/polkit-1/localauthority’: Permission denied
/etc/cron.daily/apt-compat
  • Example 3: Find all directories owned by testuser from /home directory and its subdirectories
testuser@ubuntu-pc:~$ find /home -user testuser -type d
/home/testuser
/home/testuser/.cache
/home/testuser/data
/home/testuser/alias-test
/home/testuser/.gnupg
/home/testuser/.gnupg/private-keys-v1.d
/home/testuser/.local
/home/testuser/.local/share
/home/testuser/.local/share/nano
/home/testuser/sqlit
/home/testuser/sqlit/sqlite-autoconf-3290000
/home/testuser/sqlit/sqlite-autoconf-3290000/tea
/home/testuser/sqlit/sqlite-autoconf-3290000/tea/doc
/home/testuser/sqlit/sqlite-autoconf-3290000/tea/win
/home/testuser/sqlit/sqlite-autoconf-3290000/tea/tclconfig
/home/testuser/sqlit/sqlite-autoconf-3290000/tea/generic
  • Example 4: Find all empty files and directories from /tmp directory
testuser@ubuntu-pc:~$ find /tmp -empty
/tmp/.X11-unix
/tmp/.Test-unix
/tmp/.font-unix
/tmp/.XIM-unix
/tmp/.ICE-unix
  • Example 5: Find all files and directories having at least 10000000 bytes from /tmp directory
testuser@ubuntu-PC:~$ find /home/testuser -size +10000000c
/home/testuser/sqlit/sqlite-autoconf-3290000/sqlite3-sqlite3.o
/home/testuser/sqlit/sqlite-autoconf-3290000/sqlite3.o
/home/testuser/sqlit/sqlite-autoconf-3290000/.libs/libsqlite3.a
/home/testuser/sqlit/sqlite-autoconf-3290000/.libs/sqlite3.o
  • Example 6: Find all directories with permission setting of 755 from /home directory
testuser@ubuntu-PC:~$ find /home -type d -perm 755
/home/home/testuser/home/testuser/sqlit/sqlite-autoconf-3290000
/home/testuser/sqlit/sqlite-autoconf-3290000/.deps
/home/testuser/sqlit/sqlite-autoconf-3290000/tea
/home/testuser/sqlit/sqlite-autoconf-3290000/tea/doc
/home/testuser/sqlit/sqlite-autoconf-3290000/tea/win
/home/testuser/sqlit/sqlite-autoconf-3290000/tea/tclconfig
/home/testuser/sqlit/sqlite-autoconf-3290000/tea/generic
/home/testuser/sqlit/sqlite-autoconf-3290000/.libs