Intermediate Linux Tutorial: Pipes, Redirection, awk, sed, sort, uniq, and find

Tengeeks

10 min read

·

Oct 24, 2026

--

Listen

Share

Master essential Linux command-line tools with practical examples, step-by-step explanations, and hands-on exercises.

Linux becomes much more powerful when you learn how to combine commands instead of running each command separately. With pipes, redirection, and text-processing utilities, you can analyze logs, filter information, find files, generate reports, and automate routine system administration tasks.

In this guide, we will explore seven essential Linux command-line techniques:

• Pipes (|)
• Input and output redirection
• awk
• sed
• sort
• uniq
• find

Whether you use Ubuntu, Debian, Linux Mint, Fedora, Rocky Linux, or another Linux distribution, these techniques will help you work more efficiently in the terminal.

1. Prepare Your Linux Practice Environment

Before learning the commands, create a safe workspace and a sample employee file.

Open your terminal and execute:

mkdir -p ~/linux-intermediate
cd ~/linux-intermediate

cat > employees.txt <<'EOF'
101 Amit IT 45000
102 Priya HR 38000
103 Rahul IT 52000
104 Neha Sales 41000
105 Ravi IT 45000
106 Sara HR 42000
107 Aman Sales 39000
EOF

Display the file:

cat employees.txt

You should see seven employee records, with each field separated by spaces.

The fields represent employee ID, name, department, and salary. We will use this file to practice filtering, sorting, and extracting data.

2. Understand Linux Pipes (|)

A pipe connects two commands by sending the standard output of the first command to the standard input of the second.

The general syntax is:

command1 | command2

For example:

ls -l | less

The ls -l command lists directory contents in long format, while less lets you browse the results page by page.

Filter command output

Display only employee records from the IT department:

grep ' IT ' employees.txt

Count the IT employee records:

grep -c ' IT ' employees.txt

Expected output:

3

You can also connect multiple commands into a pipeline:

grep ' IT ' employees.txt | sort -k4,4n | head -n 2

This command:

1. Selects IT employee records.
2. Sorts them numerically by salary.
3. Displays the first two records.

Expected output:

101 Amit IT 45000
105 Ravi IT 45000

Both employees have the same salary, so their relative order may vary.

Practical tip: Pipes are useful when examining server logs, filtering running processes, counting records, and preparing data for reports. Avoid unnecessary commands such as cat employees.txt | grep IT when you can simply run grep IT employees.txt.

3. Master Linux Input and Output Redirection

Linux commands usually work with three standard streams:

StreamFile descriptorPurposeStandard input0Receives inputStandard output1Produces normal outputStandard error2Produces error messages

Redirection lets you send these streams to files or other destinations.

Redirect output with >

Save a directory listing to a file:

ls -l > directory-list.txt

The > operator creates the file if necessary and overwrites its previous contents if it already exists.

Read the saved output:

cat directory-list.txt

Append output with >>

Append the current date to a report:

date >> daily-report.txt

Run the command again and it will add another line rather than replacing the earlier entry.

Remember:

• > overwrites a file.
• >> appends to a file.

Redirect error messages with 2>

Try to list a file that does not exist:

ls missing-file.txt 2> error.log

The error is saved in error.log instead of being displayed as normal terminal output.

View the error:

cat error.log

Save normal output and errors separately

find /etc -maxdepth 1 -type f > files.txt 2> errors.txt

The file listing is saved to files.txt, and error messages are directed to errors.txt.

Redirect both streams into one file

In Bash, use:

find /etc -maxdepth 1 -type f > results.log 2>&1

The order matters: 2>&1 sends standard error to the destination currently used by standard output.

Display output and save it with tee

ls -l | tee directory-list.txt

The tee command displays the output and saves it to a file simultaneously.

To append instead of overwrite:

date | tee -a directory-list.txt

This is especially useful when you want to monitor a command while keeping a record for troubleshooting.

4. Learn the Linux awk Command

awk is a powerful text-processing language. It can extract columns, filter records, calculate totals, and produce formatted reports.

Its basic structure is:

awk 'pattern { action }' filename

By default, awk separates fields using whitespace.

Useful built-in variables include:

• $1: first field
• $2: second field
• $0: entire line
• NR: current record number
• NF: number of fields

Extract columns

Display employee names:

awk '{print $2}' employees.txt

Display names and departments:

awk '{print $2, $3}' employees.txt

Display names and salaries:

awk '{print $2, $4}' employees.txt

Filter records with conditions

Find employees whose salary is greater than 40,000:

awk '$4 > 40000 {print $2, $4}' employees.txt

Find employees in the IT department:

awk '$3 == "IT" {print $2, $3, $4}' employees.txt

Find salaries of at least 45,000:

awk '$4 >= 45000 {print $2, $4}' employees.txt

awk evaluates the condition for each record and executes the action only when that condition is true.

Calculate total and average salaries

Calculate the total salary:

awk '{sum += $4} END {print "Total salary:", sum}' employees.txt

Calculate the average:

awk '{
    sum += $4
    count++
}
END {
    if (count > 0)
        print "Average salary:", sum / count
}' employees.txt

Count all employee records:

awk 'END {print "Total employees:", NR}' employees.txt

These commands are useful when analyzing reports or calculating statistics from structured text.

Process comma-separated data

Create a simple CSV file:

cat > sales.csv <<'EOF'
Name,Department,Sales
Amit,IT,50000
Priya,HR,38000
Rahul,IT,62000
EOF

Extract names and sales values:

awk -F',' 'NR > 1 {print $1, $3}' sales.csv

The -F',' option sets the field separator to a comma. The NR > 1 condition skips the header row.

This example works for simple CSV data. Files containing quoted commas or complex CSV escaping should be processed with a CSV-aware utility or programming language.

5. Learn the Linux sed Command

sed, or stream editor, processes text and commonly performs substitutions, deletes lines, and prints selected sections.

Its basic syntax is:

sed 'instruction' filename

Create a sample file:

echo "Linux is useful. Linux is powerful." > notes.txt

Replace text

Replace the first occurrence of Linux on each line:

sed 's/Linux/Ubuntu/' notes.txt

Output:

Ubuntu is useful. Linux is powerful.

Replace every occurrence on each line:

sed 's/Linux/Ubuntu/g' notes.txt

Output:

Ubuntu is useful. Ubuntu is powerful.

The s represents substitution, while g applies the replacement to every match on a line.

Save edited output safely

Preview a replacement without changing the original:

sed 's/Linux/Ubuntu/g' notes.txt

Save the result to another file:

sed 's/Linux/Ubuntu/g' notes.txt > notes-updated.txt

On GNU/Linux, you can edit in place:

sed -i.bak 's/Linux/Ubuntu/g' notes.txt

This creates a backup named notes.txt.bak before modifying the original file.

Always preview substitutions before applying them to important configuration files.

Delete selected lines

Create a sample list:

cat > colors.txt <<'EOF'
red
blue
green
blue
yellow
EOF

Delete lines containing blue:

sed '/blue/d' colors.txt

Delete the first line:

sed '1d' colors.txt

Delete lines two through four:

sed '2,4d' colors.txt

Print selected lines

Print line three:

sed -n '3p' colors.txt

Print lines two through four:

sed -n '2,4p' colors.txt

The -n option suppresses automatic printing, while p prints the selected lines.

6. Understand the Linux sort Command

The sort command arranges lines alphabetically, numerically, in reverse order, or according to selected fields.

Create a test file:

cat > numbers.txt <<'EOF'
50
10
200
30
5
EOF

Sort alphabetically

sort numbers.txt

Because the default sort treats lines as text, the order may not match numerical order.

Sort numerically

sort -n numbers.txt

Expected output:

5
10
30
50
200

Sort in descending numerical order:

sort -nr numbers.txt

Save a sorted result:

sort -n numbers.txt -o numbers-sorted.txt

Sort employee records by salary

Sort by the fourth field:

sort -k4,4n employees.txt

The -k4,4 option selects the fourth field, and -n sorts it numerically.

Sort from highest salary to lowest:

sort -k4,4nr employees.txt

To make ties more predictable, add the employee name as a secondary key:

sort -k4,4nr -k2,2 employees.txt

Find large disk-usage entries

du -ah . | sort -hr | head -n 10

This displays the ten largest reported entries under the current directory. The -h option lets sort understand size suffixes such as K, M, and G.

7. Master the Linux uniq Command

The uniq command identifies or removes adjacent duplicate lines. It does not automatically identify every repeated line scattered throughout a file.

Create a sample file:

cat > visitors.txt <<'EOF'
Amit
Priya
Amit
Rahul
Priya
Priya
Amit
EOF

Remove adjacent duplicates

uniq visitors.txt

Only consecutive duplicate lines are collapsed. Repeated names separated by other records remain.

Find unique lines throughout the file

Combine sort and uniq:

sort visitors.txt | uniq

Or use:

sort -u visitors.txt

Sorting groups identical lines together so duplicates can be removed.

If you want to remove duplicates while preserving the order of first appearance, use:

awk '!seen[$0]++' visitors.txt

Count occurrences

sort visitors.txt | uniq -c

Example output:

3 Amit
      2 Priya
      1 Rahul

Sort by frequency:

sort visitors.txt | uniq -c | sort -nr

This technique is useful for counting repeated log messages, usernames, IP addresses, and other recurring values.

Display only duplicates

sort visitors.txt | uniq -d

Display lines that occur exactly once:

sort visitors.txt | uniq -u

The input should be sorted or already grouped by identical lines to reliably detect all duplicates.

8. Master the Linux find Command

The find command searches directory trees using conditions such as filename, type, size, modification time, and permissions.

Its general syntax is:

find [starting-path] [conditions] [actions]

Find files by name

Find a file named notes.txt under the current directory:

find . -type f -name "notes.txt"

Find all text files:

find . -type f -name "*.txt"

Find directories named backup:

find . -type d -name "backup"

Search without case sensitivity:

find . -type f -iname "*.TXT"

Here, . means the current directory, -type f selects regular files, and -type d selects directories.

Find files by size

Find files larger than 100 MB:

find /var -type f -size +100M 2>/dev/null

Find files smaller than 1 MB:

find . -type f -size -1M

These examples use GNU find size syntax, which is commonly available on Linux systems.

Find recently modified files

Find files modified within the last 60 minutes:

find . -type f -mmin -60

Find files more than seven days old:

find . -type f -mtime +7

The -mtime test uses rounded-down whole 24-hour periods, so -mtime +7 generally selects files at least eight full 24-hour periods old. Use -mmin or timestamp-based tests if you need finer precision.

Find files by permissions

Find files with permissions exactly 644:

find . -type f -perm 0644

Find files writable by others:

find . -type f -perm -0002

Review permission results carefully before making changes. The appropriate permissions depend on each file’s purpose.

Find empty files and directories

Find empty files:

find . -type f -empty

Find empty directories:

find . -type d -empty

Find and process files

Display matching log files with their sizes:

find . -type f -name "*.log" -exec ls -lh {} +

The {} placeholder represents matching file paths. The + terminator allows multiple paths to be passed to each invocation.

Count the number of matching files:

find . -type f -name "*.log" | wc -l

Delete files carefully

First, preview matching temporary files:

find . -type f -name "*.tmp" -print

After verifying the results, delete them:

find . -type f -name "*.tmp" -delete

Warning: The -delete action permanently removes matching files. Always inspect the results and verify your starting directory before running destructive commands.

9. Combine These Commands in Real-World Tasks

The biggest advantage of Linux is that these tools work together.

Example 1: Find the most frequent log messages

sort application.log | uniq -c | sort -nr | head -n 10

This sorts the log lines, counts identical entries, ranks them by frequency, and displays the ten most common lines.

Example 2: Count HTTP 5xx errors

For a standard whitespace-separated web access log where the HTTP status code is field nine:

awk '$9 ~ /^5[0-9][0-9]$/ {print $9}' access.log |
sort |
uniq -c |
sort -nr

This extracts server-error status codes, counts each code, and ranks the totals.

Your actual field positions may differ if you use a custom log format.

Example 3: Generate an IT salary report

awk '$3 == "IT" {print $2, $4}' employees.txt |
sort -k2,2nr > it-salary-report.txt

View the report:

cat it-salary-report.txt

This command selects IT employees, extracts their names and salaries, sorts by salary in descending order, and saves the result.

Example 4: Locate large log files

find /var/log -type f -name "*.log" -size +10M \
    -exec ls -lh {} + > large-log-files.txt 2> find-errors.log

The command writes the listing to large-log-files.txt and errors to find-errors.log.

Use the output to identify large logs that may need rotation, archiving, or investigation.

10. Practice Exercises

Try these tasks in your Linux practice directory.

Exercise 1: Count the records in employees.txt.

wc -l < employees.txt

Exercise 2: Print only employee names.

awk '{print $2}' employees.txt

Exercise 3: Display employees earning more than 40,000.

awk '$4 > 40000 {print $2, $4}' employees.txt

Exercise 4: Replace Ubuntu with Linux in notes.txt without changing the original file.

sed 's/Ubuntu/Linux/g' notes.txt

Exercise 5: Sort employees from highest salary to lowest.

sort -k4,4nr employees.txt

Exercise 6: Count each name in visitors.txt.

sort visitors.txt | uniq -c

Exercise 7: Find configuration files below /etc.

find /etc -type f -name "*.conf" 2>/dev/null

Exercise 8: Save and display a directory listing at the same time.

ls -l | tee report.txt

Run each command, study the output, and change the examples to understand how each option works.

11. Common Linux Command-Line Mistakes

uniq does not remove every duplicate

Cause: Duplicate lines are not adjacent.

Fix:

sort file.txt | uniq

sort -n produces unexpected results

Cause: The input may contain text, or the numeric value may be in another field.

Fix: Inspect the data and select the correct field with -k.

awk displays the wrong column

Cause: The assumed field separator does not match the input.

Fix: Use -F to set a separator, such as awk -F',' for simple comma-separated data.

sed -i modifies a file unexpectedly

Cause: A replacement pattern may be too broad.

Fix: Preview the output first and create a backup before editing important files.

find reports permission errors

Cause: Your account cannot access some directories.

Fix: Search accessible paths or redirect errors to a separate file. Do not use elevated privileges unless the task genuinely requires them.

A pipeline hides an earlier failure

In Bash, enable pipeline failure detection when needed:

set -o pipefail

This helps a pipeline return a failure status when an earlier command fails. It is particularly useful in shell scripts.

Conclusion

Intermediate Linux commands are essential tools for anyone learning Linux system administration, DevOps, shell scripting, or server troubleshooting.

Pipes and redirection connect commands and control their output. awk extracts fields and calculates values, while sed edits streams of text. sort organizes records, uniq identifies repeated lines, and find locates files across directory trees.

The best way to learn these commands is to practice them with real files, examine their outputs, and gradually combine them into repeatable workflows.

Continue building your skills with practical tutorials and technology guides on TenGeeks.

Keep practicing, experiment safely, and make the Linux command line work for you.