Shell脚本题库

Shell 题库脚本

1、统计文件的行数

编写一个shell脚本以输出一个文本文件nowcoder.txt中的行数

1
2
3
4
5
6
7
8
#!/bin/bash
1 grep -n "" nowcoder.txt|awk -F: '{print }'|tail -n1| awk -F: '{print $1}'
2 awk '{print NR}' nowcoder.txt|tail -n1
3 awk 'END{print NR}' nowcoder.txt
4 sed -n '$=' nowcoder.txt
5 cat nowcoder.txt |wc -l

## 以上5种方法均可

2、打印文件的最后5行

查看日志的时候，经常会从文件的末尾往前查看，请你写一个bash shell脚本以输出一个文本文件nowcoder.txt中的最后5行

1
2
3
4
5
6
7
8
9
10
11
12
#!/bin/bash
tail -n 5 nowcoder.txt

## tail 命令可用于查看文件的内容，有一个常用的参数 -f 常用于查阅正在改变的日志文件。
 #   -f 循环读取
 #   -q 不显示处理信息
 #   -v 显示详细的处理信息
 #   -c<数目> 显示的字节数
 #   -n<行数> 显示文件的尾部 n 行内容
 #   --pid=PID 与-f合用,表示在进程ID,PID死掉之后结束
 #   -q, --quiet, --silent 从不输出给出文件名的首部
 #   -s, --sleep-interval=S 与-f合用,表示在每次反复的间隔休眠S秒

3、输出 0 到 500 中 7 的倍数

写一个 bash脚本以输出数字 0 到 500 中 7 的倍数(0 7 14 21…)的命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/bin/bash
1 seq 0 7 500
2 
for i in {0..500..7}; do 
  echo  $i
done
3 seq 0 500|awk '$0%7==0 {print $0}' 
4 seq 0 500 | sed -n '1~7p'

# 以上4种方法均可
# seq 用于生成从一个数到另一个数之间的所有整数。
# 用法：seq [选项]... 尾数
# 或：seq [选项]... 首数 尾数
# 或：seq [选项]... 首数 增量 尾数

4、输出文本文件的第5行的内容

编写一个bash脚本以输出一个文本文件nowcoder.txt中第5行的内容

1
2
3
4
5
6
7
8
9
10
11
12
#!/bin/bash
1 sed -n 5p		or		sed -n '5p'
2 awk 'NR==5{print $0}'
3 head -n 5 nowcoder.txt | tail -n 1
4 sed -n  -e "5,5p" nowcoder.txt

# 以上4种方法均可
# 如果你只想看文件的前100行，可以使用head命令，如	head -100 $n0
# 如果你想查看文件的后100行，可以使用tail命令，如： tail -100 $n0 或 tail -n 100 $n0
# 查看文件中间一段，你可以使用sed命令，如：
# sed -n ‘100,200p’ $n0
# 这样你就可以只查看文件的第100行到第200行

5、打印空行的行号

编写一个shell脚本以输出一个文本文件nowcoder.txt中空行的行号（空行可能连续，从1开始输出）

1
2
3
4
5
6
7
8
9
10
#!/bin/bash
1 awk '/^$/ {print NR}'
2 awk '{if($0 == "") {print NR}}' ./nowcoder.txt
3 awk  'NF==0{print NR}'  nowcoder.txt
4 awk '/^\s*$/{print NR}' nowcoder.txt
5 sed -n  '/^$/='  "nowcoder.txt" 
#-n 对匹配的行做处理 =打印匹配到的内容的行号 p打印匹配到的内容
6 grep -n '^$' nowcoder.txt |cut -d ':' -f 1
7 grep -n '^$' nowcoder.txt | awk -F: '{print $1}'
8 grep -n '^$' nowcoder.txt | sed 's/\:/\ /g'

6、去掉空行

写一个 bash脚本以去掉一个文本文件nowcoder.txt中的空行

1
2
3
4
5
6
7
8
9
10
11
12
1 grep . 
2 grep -E  '\S+'
3 grep -v '^$' nowcoder.txt
4 sed -n '/[^$]/p'
5 sed '/^$/d' nowcoder.txt
6 cat nowcoder.txt | awk NF
7 awk NF
8 awk '!/^$/{print $NF}' 
#p：打印变动的流（行）
#正则部分： [^$] ^代表以后面跟着的字符为开头，$代表以前面的字符为结尾；
#^$联合使用，中间不加任何字符数字，代表匹配空行；
#[ ] 在shell正则中表示取反

7、打印字母数小于8的单词

写一个bash脚本以统计一个文本文件nowcoder.txt中字母数小于8的单词

1
2
3
4
# 1、 用空格进行分割，NF是当前记录的字段数，也可以说是单词数;然后for循环嵌套if判断;
#     当当前字段的长度小于8时，将其打印出来;
#!/bin/bash
awk -F" " '{for(i=1;i<=NF;i++){if(length($i) < 8){print $i}}}' nowcoder.txt
1
2
3
# 2、先使用 xargs 命令将输出转换为单行，然后指定 awk 的分隔符为 ""，设定每行的筛选条件为 NF<8
#!/bin/bash
cat nowcoder.txt | xargs -n1 | awk -F "" 'NF<8'
1
2
3
# 3、在这个脚本中，grep 命令使用 -o 参数来只输出匹配的部分，使用 -E 参数启用扩展的正则表达式。正则表达式 \b\w{1,7}\b 匹配字母数小于8的单词，其中 \b 表示单词边界，\w 匹配字母数字字符，{1,7} 表示匹配长度为1到7的单词。
# 运行这个脚本后，它将输出所有字母数小于8的单词。
grep -oE '\b\w{1,7}\b' nowcoder.txt
1
2
3
4
5
6
7
8
9
# 4、
#!/bin/bash
words=$(cat nowcoder.txt)
for word in ${words}
do
    if [ ${#word} -lt 8 ];then
       echo ${word}
    fi
done
1
2
3
4
5
6
7
# 5、
#!/bin/bash
for i in $(cat nowcoder.txt);do
if [[ ${#i} -lt 8 ]] ; then
echo $i
fi
done

8、统计所有进程占用内存百分比的和

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
假设 nowcoder.txt 内容如下：
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.4  77744  8332 ?        Ss    2021   1:15 /sbin/init noibrs splash
root         2  0.0  0.0      0     0 ?        S     2021   0:00 [kthreadd]
root         4  0.0  0.0      0     0 ?        I<    2021   0:00 [kworker/0:0H]
daemon     486  0.0  0.1  28340  2372 ?        Ss    2021   0:00 /usr/sbin/atd -f
root       586  0.0  0.3  72308  6244 ?        Ss    2021   0:01 /usr/sbin/sshd -D
root     12847  0.0  0.0   4528    68 ?        S<   Jan03   0:13 /usr/sbin/atopacctd
root     16306  1.7  1.2 151964 26132 ?        S<sl Apr15 512:03 /usr/local/aegis/aegis_client/aegis_11_25/AliYunDun
root     24143  0.0  0.4  25608  8652 ?        S<Ls 00:00   0:03 /usr/bin/atop -R -w /var/log/atop/atop_20220505 600
root     24901  0.0  0.3 107792  7008 ?        Ss   15:37   0:00 sshd: root@pts/0
root     24903  0.0  0.3  76532  7580 ?        Ss   15:37   0:00 /lib/systemd/systemd --user
root     24904  0.0  0.1 111520  2392 ?        S    15:37   0:00 (sd-pam)
以上内容是通过ps aux命令输出到nowcoder.txt文件中的，请你写一个脚本计算一下所有进程占用内存大小的和。

示例1
输入：
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.4  77744  8332 ?        Ss    2021   1:15 /sbin/init noibrs splash
root         2  0.0  0.0      0     0 ?        S     2021   0:00 [kthreadd]
root         4  0.0  0.0      0     0 ?        I<    2021   0:00 [kworker/0:0H]
daemon     486  0.0  0.1  28340  2372 ?        Ss    2021   0:00 /usr/sbin/atd -f
root       586  0.0  0.3  72308  6244 ?        Ss    2021   0:01 /usr/sbin/sshd -D
root     12847  0.0  0.0   4528    68 ?        S<   Jan03   0:13 /usr/sbin/atopacctd
root     16306  1.7  1.2 151964 26132 ?        S<sl Apr15 512:03 /usr/local/aegis/aegis_client/aegis_11_25/AliYunDun
root     24143  0.0  0.4  25608  8652 ?        S<Ls 00:00   0:03 /usr/bin/atop -R -w /var/log/atop/atop_20220505 600
root     24901  0.0  0.3 107792  7008 ?        Ss   15:37   0:00 sshd: root@pts/0
root     24903  0.0  0.3  76532  7580 ?        Ss   15:37   0:00 /lib/systemd/systemd --user
root     24904  0.0  0.1 111520  2392 ?        S    15:37   0:00 (sd-pam)
输出：3.1
1
2
3
4
5
6
7
8
9
# 方法1： 使用awk进行运算比较好, 因为shell中本身是没有浮点运算的，只能借助awk或者bc工具
# 先把初始化写好 BEGIN{sum=0.0} # 初始化变量
# 再把最终的输出写好 END{print sum} # 最后打印变量
# 然后过滤出符合条件的行, 因为第一行不能用来计算： if (NR>1) {#这里面处理} ---简化后---> NR>1
# 然后将第4列的数据累加起来也就是：{sum+=$4}
# 写全一点就是：
awk 'BEGIN{sum=0.0} {if(NR>1){sum+=$4}} END{print sum}' nowcoder.txt
# 简化后是：
awk 'BEGIN{sum=0.0} NR>1 {sum+=$4} END{print sum}' nowcoder.txt
1
2
3
4
5
6
7
8
#方法2：
awk '{print $4}'| awk 'NR>1 {sum+=$1} END {print sum}'

#方法3：
awk '{rss+=$4}END{print rss}' nowcoder.txt

#方法4：
awk '{num=num+$4}END{print num}' nowcoder.txt

9、统计每个单词出现的个数

写一个bash脚本以统计一个文本文件nowcoder.txt 中每个单词出现的个数。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# 方法1：
tr ' ' '\n' | sort | uniq -c | awk '{print $2, $1}' | sort -nk2

#方法2：
cat nowcoder.txt | awk '{for(i=1;i<=NF;i++){print $i}}'|sort|uniq -c| sort -n | awk '{print $2,$1}'

#方法3：
cat nowcoder.txt | tr -s ' ' '\n' | sort | uniq -c | awk '{print $2" "$1}' | sort -k2n

#方法4：
xargs -n1 | sort | uniq -c | sort | awk '{print $2" "$1}'

#方法5：
awk '{
    for(i = 1 ; i <= NF ; i++){
        array[$i] += 1; 
    }
}
END{
    for(s in array){
        print s" "array[s];
    }
}' nowcoder.txt | sort -k2n
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#方法6：

#简洁做法：
#!/bin/bash
# 检查文件 nowcoder.txt 是否存在
if [ ! -f "nowcoder.txt" ]; then
    echo "File nowcoder.txt does not exist."
    exit 1
fi

# 使用 tr 将空格转换为换行符，这样每个单词都在新的一行
# 使用 sort 对单词进行排序，使相同的单词排在一起
# 使用 uniq -c 统计每个单词的出现次数
# 再次使用 sort 对输出进行排序，先按数字排序
cat nowcoder.txt | tr -s ' ' '\n' | sort | uniq -c | sort -n | awk '{print $2, $1}'
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#高效实现：
#!/bin/bash
# 确保 nowcoder.txt 文件存在
if [ ! -f "nowcoder.txt" ]; then
    echo "File nowcoder.txt does not exist."
    exit 1
fi

awk '{
    for (i = 1; i <= NF; i++) {
        freq[$i]++
    }
}
END {
    for (word in freq) {
        print word, freq[word]
    }
}' nowcoder.txt | sort -k2,2n -k1,1

10、第二列是否有重复

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
给定一个nowcoder.txt文件，其中有3列信息，如下：
20201001 python 99
20201002 go 80
20201002 c++ 88
20201003 php 77
20201001 go 88
20201005 shell 89
20201006 java 70
20201008 c 100
20201007 java 88
20201006 go 97
编写一个shell脚本来检查文件第二列是否有重复，且有几个重复，并提取出重复的行的第二列信息（先按次数排序，如果次数相同，按照单词字母顺序排序），输入如下：
2 java
3 go
1
2
3
4
5
6
1 awk '{print $2}' nowcoder.txt | sort | uniq -cd | sort -n
2 cat  $1 |awk '{print $2}'  |sort  |uniq -c|sort |grep -v 1
3 
#!/bin/bash
awk '{print $2}' nowcoder.txt | sort | uniq -c | sort -nk1,1 -k2,2 | awk '$1>1{print $1,$2}'
4 cat nowcoder.txt | sort -k 2 | cut -d " " -f 2 | uniq -c | awk '{if ($1 > 1) {print $0}}' | sort

11、转置文件的内容

描述

1
2
3
4
5
6
7
8
9
10
11
12
写一个bash脚本来转置文本文件nowcoder.txt中的文件内容。
文件中每行列数相同，并且每个字段由空格分隔
示例:
假设 nowcoder.txt 内容如下：
job salary
c++ 13
java 14
php 12

你的脚本应当输出（以词频升序排列）：
job c++ java php
salary 13 14 12
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#方法1：
#!/bin/bash
a=$( cat nowcoder.txt | awk '{print $1}' )
b=$( cat nowcoder.txt | awk '{print $2}' )
echo $a
echo $b

#方法2(shell)：
awk '{
    for(i=1;i<=NF;i++){
      if(NR==1){
        row[i] = $i;
      }else{
        row[i] = row[i]" "$i;
      }
    }
}END{
  for(i=1;i<=NF;i++){
    print row[i]
  }
}
' ./nowcoder.txt

12、打印每一行出现的数字个数

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
写一个bash脚本，统计一个文本文件nowcoder.txt中每一行出现的1~5数字的个数，并且计算一下整个文档中一共出现了几个1~5数字的总数。
示例:
假设 nowcoder.txt 内容如下：
a12b8
10ccc
2521abc
9asf
你的脚本应当输出：
line1 number: 2
line2 number: 1
line3 number: 4
line4 number: 0
sum is 7
1
2
3
4
5
6
7
8
9
10
11
12
#方法1：
#!/bin/bash
total=0
i=0
while read -r line; do
  let i++
  sum=$(echo "$line" |grep -o [1-5] |wc -l)
  echo "line$i number:$sum"
  total=$((total + sum))
done <nowcoder.txt

echo "sum is $total"
1
2
#方法2：
awk -F "[1,2,3,4,5]" 'BEGIN{sum=0}{print "line"NR" number:"(NF-1);sum+=(NF-1)}END{print "sum is "sum}' nowcoder.txt

13、去掉所有包含this的句子

编写一个shell脚本以实现如下功能：去掉输入中含有this的语句，把不含this的语句输出

1
2
3
4
1 grep -v 'this'
2 sed '/this/d'
3 sed -n '/this/!p'  nowcoder.txt
4 cat $1 |grep -v "this"

14、求平均值

描述

1
2
3
4
5
6
7
8
9
10
11
写一个bash脚本以实现一个需求，求输入的一个数组的平均值

第1行为输入的数组长度N
第2~N行为数组的元素，如以下为:
数组长度为4，数组元素为1 2 9 8
示例:
4
1
2
9
8
1
2
3
1 awk -v line=0 -v sum=0 -v arg=0 'NR!=1{sum=sum+$1} NR==1{line=$1} END{printf "%.3f", arg=sum/line}'
2 xargs|awk '{for(i=2;i<=NF;i++){sum+=$i}}END{printf ("%.3f",sum/$1)}'
3 awk 'BEGIN{sum=0;}{if(NR!=1){sum+=$1;}}END{printf("%.3f\n",sum/(NR-1))}' nowcoder.txt

15、去掉不需要的单词

描述

1
2
3
4
5
6
7
8
9
10
11
写一个bash脚本以实现一个需求，去掉输入中含有B和b的单词
示例:
假设输入如下：
big
nowcoder
Betty
basic
test

你的脚本获取以上输入应当输出：
nowcoder test
1
2
3
4
5
6
7
8
9
1 grep -v -E 'b|B' nowcoder.txt
2 grep -iv "b"
3 grep -v '[bB]' 
4 cat nowcoder.txt | grep -v -E 'b|B' 
5 cat nowcoder.txt|grep -vi "b"
6 awk '$0!~/b|B/ {print $0}' nowcoder.txt
7 awk '!/[bB]/'
8 sed '/[Bb]/d'
9 sed '/b\|B/d'

16、判断输入的是否为IP地址

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
写一个脚本统计文件nowcoder.txt中的每一行是否是正确的IP地址。
如果是正确的IP地址输出：yes
如果是错误的IP地址，且是四段号码的话输出：no，否则的话输出：error
假设nowcoder.txt内容如下：
192.168.1.1
192.168.1.0
300.0.0.0
123

你的脚本应该输出：
yes
yes
no
error
1
2
3
4
5
6
7
8
9
10
11
12
# 纯shell:
while read line; do
	arr=(${line//./ })
	if [ ${#arr[*]} -ne 4 ]; then
		printf "error\n"
	else
		for ((i = 0; i < ${#arr[*]}; i++)); do
			[ ${arr[${i}]} -gt 255 ] && printf "no\n" && break
		done
		[ $i == 4 ] && printf "yes\n"
	fi
done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# 纯awk:
awk -F "." '{
    if (NF == 4) {
        for (i=1; i<5; i++) {
            if ($i > 255 || $i < 0) {
                print("no");break
            }
        }
        if (i==5){print("yes")}
    } else {
        print("error")
    }
}'

# 纯awk 正则匹配版本:
awk -F "." '{
    if (NF == 4) {
        if ($0 ~ /^((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[09][0-9]|[1-9][0-9]|[0-9])$/) {
                print("yes")
        } else {
                print("no")
        }
    } else {
        print("error")
    }
}'

17、将字段逆序输出文件的每行

描述

1
2
3
4
5
6
7
编写一个shell脚本，将文件nowcoder.txt中每一行的字段逆序输出，其中字段之间使用英文冒号:相分隔。
假设nowcoder.txt内容如下：
nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
root:*:0:0:System Administrator:/var/root:/bin/sh
你的脚本应当输出
/usr/bin/false:/var/empty:Unprivileged User:-2:-2:*:nobody
/bin/sh:/var/root:System Administrator:0:0:*:root
1
2
3
4
5
6
7
8
#!/bin/bash
awk -F ':' '{
    sum = $NF
    for (i=NF-1; i>=1; i--){
        sum = sum ":" $i
    }
    print sum
}'

18、域名进行计数排序处理

描述

1
2
3
4
5
6
7
8
假设有一些域名，存储在nowcoder.txt里，现在需要写一个shell脚本，将域名取出并根据域名进行计数排序处理（降序）。
假设nowcoder.txt内容如下：
http://www.nowcoder.com/index.html
http://www.nowcoder.com/1.html
http://m.nowcoder.com/index.html
你的脚本应该输出：
2 www.nowcoder.com
1 m.nowcoder.com
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#方法一：
#!/bin/bash
: <<EOF

EOF

cat nowcoder.txt | awk -F "[/]+" '{
    let arr[$2]++; #对第二列每个字符串出现的次数进行统计
}END{
    for(k in arr){
        printf "%d %s\n",arr[k],k;
    }
}' | sort -k 1 | awk '{arr[NR] = $0}
END{
    for(i = NR;i >=1;i--) print arr[i];
}'
1
2
3
4
5
6
7
8
9
10
11
12
#方法二：
#!/bin/bash
: <<EOF
EOF

cat nowcoder.txt | awk -F "[/]+" '{
    let arr[$2]++; #对第二列每个字符串出现的次数进行统计
}END{
    for(k in arr){
        printf "%d %s\n",arr[k],k;
    }
}' | sort -r
1
2
3
4
5
#方法3：获取第二列域名 + sort排序 + uniq -c计算重复次数 + sort按第一列降序排序 + awk调整输出格式

#!/bin/bash
cat nowcoder.txt | awk -F "[/]+" '{print $2}' | sort | uniq -c | sort -r | awk -F "[ ]+" '{
    printf "%d %s\n",$2,$3;}'

19、打印等腰三角形

描述

1
2
3
4
5
6
7
8
9
编写一个shell脚本，输入正整数n，打印边长为n的等腰三角形。
示例：
输入：5
输出：
    *
   * *
  * * *
 * * * *
* * * * *
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#方法一：
while read line; do
    i=1
    while [ $i -le $line ]; do
        j=1
        while [ $j -le $(($line-$i)) ]; do
            echo -n " "
            j=$(($j+1))
        done
        for ((k=1;k<=$i;k++)); do
            echo -n "* "
        done
        i=$(($i+1))
        echo ""
    done
done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#方法二、三：
#方法二：最笨的方法
# 1： 打印4个空格+"* "*1
# 2:  打印3个空格+"* "*2
# 3:  打印2个空格+"* "*3
# 4:  打印1个空格+"* "*4
# 5:  打印0个空格+"* "*5
# 若打印num等边三角形，则n行应该是
# n： 打印num-n个空格+打印n个“* ”
read -r num
for n in $(seq 1 "$num"); do
    # 打印num-index个空格
    # shellcheck disable=SC2007
    for i in $(seq 1 $[$num-$n]); do
        echo -n ' '
    done
    # 打印n个“* ”
    for j in $(seq 1 "$n"); do
        echo -n '* '
    done
    # 每个循环过后回车换行
    echo
done

# 方法三：使用awk
# 考察点： 由于awk中的语法是c语言，所以num不能再awk的代码块中直接使用，因此使用awk -v {代码块中的变量名}=${shell中的变量名}
# 然后再awk中用正常的思维撰写代码即可
read -r num
awk -v num="${num}" 'BEGIN{
    for(i=0;i<num;i++){
       row=""
       for(j=1;j<num-i;j++) {
           row=row " "
       }
       for(j=0;j<=i;j++) {
           row=row "* "
      }
      print row
    }
}'

20、打印只有一个数字的行

描述

1
2
3
4
5
6
7
8
9
10
11
假设有一个nowcoder.txt，编写脚本，打印只有一个数字的行。

假设nowcoder.txt内容如下：
haha
1
2ab
cd
77
那么你的脚本应该输出
1
2ab
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#纯shell:
while read line
do
    count=0
    for ((i=0;i<${#line};i++))
        do
            [[ ${line:i:1} =~ [0-9] ]] && ((count++))
        done
    if [ $count -eq 1 ];then
        printf "$line\n"
    fi
done < nowcoder.txt

#2：
while read line; do
    out=${line//[^0-9]/}  # 进行字符串替换，去除非数字字符
 
    if [ ${#out} -eq 1 ]; then # 计算数字字符长度并输出
        echo $line
    fi
done < nowcoder.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
#awk:
awk -F "[0-9]" '{if(NF==2) print $0}' nowcoder.txt

#纯awk:
awk -F "" '{
    for (i=1; i<=NF; i++) {
        if ($i ~ /[0-9]/) {
            k++
        }
    }
    if (k==1){print($0)}
    k=0
}'

21、格式化输出

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
有一个文件nowcoder.txt，里面的每一行都是一个数字串，编写一个shell脚本对文件中每一行的数字串进行格式化：每3个数字加入一个逗号（,）。
例如：数字串为“123456789”，那么需要格式化为123,456,789。

假设nowcoder.txt内容如下：
1
12
123
1234
123456
那么你的脚本输出如下：
1
12
123
1,234
123,456
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# 方法1: 使用awk,
# 思路很简单，逐行遍历，然后倒序追加，最后就打印
awk -F "" 'BEGIN{}{
    str="";
    for(i=NF;i>=1;i--){
        if(i!=NF&&(NF-i)%3==0) {
            str=$i "," str
        } else {
            str=$i str
        }
    }
    print str
}END{}' nowcoder.txt

# 写代码的时候发现，其实不用倒序, 脑袋秀逗了~
#1
#12
#123
#1,234  # 第2次循环的时候，应该是2,后面追加',2'
#123,456
# 可以归纳为：当(NF-i)%3==0时, 只需要追加一个逗号即可
# 优化后的代码是
awk -F "" 'BEGIN{}{
    str="";
    for(i=1;i<=NF;i++){
        str=str $i
        if((NF-i)%3==0&&i!=NF) {
            str=str ","
        }
    }
    print str
}END{}' nowcoder.txt

# 方法3：使用printf, 其中%号表示占位符的开始，整型是%d, 只需要在%和d之间加个单引号即可；
#       扩展， %d默认为右对齐，%-d表示左对齐
# shellcheck disable=SC2013
for i in $(cat nowcoder.txt);do
  printf "%'-d\n" "$i"
done

22、处理文本

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
有一个文本文件nowcoder.txt，假设内容格式如下：
111:13443
222:13211
111:13643
333:12341
222:12123
现在需要编写一个shell脚本，按照以下的格式输出：
[111]
13443
13643
[222]
13211
12123
[333]
12341
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# 方法1: 第一时间想到的思路
# 首先得到所有的key： awk -F ':' '{print $1}' nowcoder.txt | sort | uniq
# 然后遍历所有的key，使用grep+awk打印所有的value
for each_key in $(awk -F ':' '{print $1}' nowcoder.txt | sort | uniq); do
    echo "[${each_key}]"
    # shellcheck disable=SC2086
    # shellcheck disable=SC2013
    for each_value in $(grep ${each_key} nowcoder.txt | awk -F ':' '{print $2}'); do
        echo ${each_value}
    done
done

# 方法2 使用字典，【注意】练手用的，不推荐，输出结果是无序的，所以测试不通过，但是其实结果是对的，供学习使用
declare -A dict
while read -r line; do
    # 获取key
    key=${line%:*}
    value=${line#*:}
    dict[${key}]=${dict[${key}]}' '$value
done < nowcoder.txt

# 遍历打印
# shellcheck disable=SC2068
for each_key in ${!dict[@]}; do
    echo "[${each_key}]"
    for each_value in ${dict[${each_key}]}; do
        echo "$each_value"
    done
done

# 方法3： 使用纯awk
# 将方法2换成使用纯awk
awk -F ':' 'BEGIN{}{
    dict[$1]=dict[$1] $2 "\n"
}END{
    for(each_key in dict){
        printf("[%s]\n", each_key)
        printf("%s", dict[each_key])
    }
}' nowcoder.txt

23、Nginx日志分析1-IP访问次数统计

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
假设 Nginx 的日志存储在 nowcoder.txt 里，内容如下：
192.168.1.20 - - [21/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:16:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在需要编写 Shell 脚本统计出 2020 年 4 月 23 号访问 IP 的对应次数，并且按照次数降序排序。你的脚本应该输出：
5 192.168.1.22
4 192.168.1.21
3 192.168.1.20
2 192.168.1.25
1 192.168.1.24
1
2
3
4
5
6
7
8
9
10
11
12
13
# 方法1：
grep '23/Apr/2020' nowcoder.txt | awk '{print $1}'   | sort  | uniq -c | sort -rk1 |awk '{print $1" "$2}'

# 方法2：
awk 'BEGIN{}{
    if(substr($4,2,11)=="23/Apr/2020"){
        dict[$1]++
    }
}END{
    for(each_ip in dict){
        printf("%d %s\n", dict[each_ip], each_ip)
    }
}' nowcoder.txt | sort -rk1
1
2
3
4
5
6
7
8
9
#grep+awk+sort:
grep "23/Apr/2020" | awk -v FS=" - - " '{
    a[$1]++
}
END{
    for(i in a){
        print(a[i]" "i)
    }
}' | sort -nr 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#希尔排序
declare -A map
while read line
    do
        tmp=($line)
        [[ ${tmp[3]} =~ 23/Apr ]] && ((map["${tmp[0]}"]++))
    done < nowcoder.txt
function ShellSort(){
    tmp=()
    for ve in ${map[*]}	#map的值转到tmp数组里
        do
            tmp[${#tmp[*]}]=$ve
        done
    q=${#tmp[*]}
	for (( gap=q>>1; gap>0; gap=gap>>1 )) #希尔排序，对tmp数组进行排序
	    do
	        for (( i=gap; i<q; i++ ))
	            do
	                temp=${tmp[$i]}
	                for (( a=i-gap; a>=0&temp>${tmp[$a]}; a-=gap ))
	                    do
	                        tmp[$a+$gap]=${tmp[$a]}
	                    done
	                tmp[$a+$gap]=$temp
	            done
	    done
}
ShellSort
for ((i=0; i<$q; i++))
    do
        for ve in ${!map[*]}
            do
                if [ ${tmp[$i]} -eq ${map[$ve]} ];then
                    printf "${map[$ve]} $ve\n"
                fi
            done
    done

24、nginx日志分析2-统计某个时间段的IP访问量

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
假设 Nginx 的日志存储在 nowcoder.txt 里，内容如下：
192.168.1.20 - - [21/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:16:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在需要编写 Shell 脚本统计 2020年04月23日20点至23点去重后的 IP 访问量，你的脚本应该输出：
5
输出说明：2020年04月23日20点至23点，共有 192.168.1.24、192.168.1.25、192.168.1.20、192.168.1.21、192.168.1.22 共 5 个 IP 访问了。
1
2
3
4
5
6
7
8
#方法1：先匹配指定时间，得到ip，先进行去重统计行数
awk '/\[23\/Apr\/2020:2[0-3]/{print $1}' nowcoder.txt | sort | uniq | wc -l

#方法2：awk直接暴力破解：
awk -F '[/ :]' '($4=="[23" && $5=="Apr" && $6==2020 && $7>=20 && $7<=23){print $1}' | sort |uniq | wc -l

#方法3：
grep -E "23/Apr/2020:2[0-3]" nowcoder.txt | awk -F " " '{print $1}'| sort |uniq -c|awk 'END{print NR}'

25、nginx日志分析3-统计访问3次以上的IP

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
假设nginx的日志我们存储在nowcoder.txt里，格式如下：
192.168.1.20 - - [21/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:16:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在需要编写shell脚本统计访问3次以上的IP，你的脚本应该输出：
6 192.168.1.22
5 192.168.1.21
4 192.168.1.20
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#shell:
declare -A map
while read line; do
	tmp=($line)
	((map["${tmp[0]}"]++))
done <nowcoder.txt
for i in ${!map[*]}; do
	[ ${map[${i}]} -le 3 ] && unset map[$i]
done
function InsertSort() {
	tmp=()
	for ve in ${map[*]}; do
		tmp[${#tmp[*]}]=$ve
	done
	q=${#tmp[*]}
	for ((i = 0; i < $q; i++)); do
		for ((j = $i + 1; j < $q; j++)); do
			if [ ${tmp[$i]} -lt ${tmp[$j]} ]; then
				t=${tmp[$i]}
				tmp[$i]=${tmp[$j]}
				tmp[$j]=$t
			fi
		done
	done
}
InsertSort
for ((i = 0; i < $q; i++)); do
	for ve in ${!map[*]}; do
		if [ ${tmp[$i]} -eq ${map[$ve]} ]; then
			printf "${map[$ve]} $ve\n"
		fi
	done
done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#awk:
#1:
awk '{
    arr[$1]++
} END {
    for (i in arr) {
        if (arr[i]>3){
            printf("%d %s\n", arr[i], i)
        }
    }
}' | sort -r
#2:
awk '{a[$1]++} END{for(i in a) if(a[i]>3) {print a[i],i}}'|sort -r
#3：
awk '{print $1}'|sort|uniq -c|awk '{if($1>3) {print $1,$2}}'|sort -r
1
2
3
4
5
6
7
8
9
10
11
#awk数组+sort：
awk -v FS=" - - " '{
    a[$1]++;
}
END{
    for(i in a){
        if(a[i] > 3){
            print(a[i]" "i)
        }
    }
}' nowcoder.txt |sort -nr

26、Nginx日志分析4-查询某个IP的详细访问情况

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
假设Nginx的日志存储在nowcoder.txt里，内容如下：
192.168.1.20 - - [21/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:16:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在需要编写shell脚本查询192.168.1.22的详细访问次数情况，按访问频率降序排序。你的脚本应该输出：
4 /1/index.php
2 /3/index.php
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/bin/bash

# 思路：
# grep 查询出 192.168.1.22
# awk 默认空格分隔的情况下    /1/index.php  在$7
# 关联数组
# 按访问频率降序排序

grep '192.168.1.22' nowcoder.txt |
	awk '
    {
        arr[$7]++
    }
    END{
        for (i in arr)
            print arr[i], i
    }
' |
	sort -k1,1r

# -k  指定 排序的列
# r   逆序排序
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#shell：
declare -A map
while read line; do
	tmp=($line)
	[ ${tmp[0]} == "192.168.1.22" ] && ((map["${tmp[6]}"]++))
done <nowcoder.txt
function InsertSort() {
	tmp=()
	for ve in ${map[*]}; do
		tmp[${#tmp[*]}]=$ve
		printf "${tmp[${#tmp[*]}]}"
	done
	q=${#tmp[*]}
	for ((i = 0; i < $q; i++)); do
		for ((j = $i + 1; j < $q; j++)); do
			if [ ${tmp[$i]} -lt ${tmp[$j]} ]; then
				t=${tmp[$i]}
				tmp[$i]=${tmp[$j]}
				tmp[$j]=$t
			fi
		done
	done
}
InsertSort
for ((i = 0; i < $q; i++)); do
	for ve in ${!map[*]}; do
		if [ ${tmp[$i]} -eq ${map[$ve]} ]; then
			echo "${map[$ve]} $ve"
		fi
	done
done
1
2
3
4
5
6
7
8
9
10
#awk:
awk '{
    if ($1 == "192.168.1.22") {
        a[$7]++
    }
} END {
    for (i in a){
        printf("%d %s\n",a[i], i)
    }
}' | sort -r

27、nginx日志分析5-统计爬虫抓取404的次数

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
假设nginx的日志存储在nowcoder.txt里，内容如下：
192.168.1.20 - - [21/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 301 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 500 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:21:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:05:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 300 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 500 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:59:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在需要编写shell脚本统计百度爬虫抓取404的次数，你的脚本应该输出
2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#!/bin/bash

# 思路:
# awk 默认空格分隔的情况下  404 在$9   百度 在$15
# if 判断 $9 == 404 和 $15 ~ /baidu/   然后 sum++

# 另一个思路：2个grep  1个wc
# grep 'baidu' | grep '404' | wc -l

awk '
    BEGIN{sum = 0}
    {
        if ($9 == 404 && $15 ~ /baidu/)
            sum++
    }
    END{
        print sum
    }
' nowcoder.txt

28、Nginx日志分析6-统计每分钟的请求数

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
假设Nginx的日志存储在nowcoder.txt里，内容如下：
192.168.1.20 - - [21/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:21:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:05:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Baiduspider"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Baiduspider"
192.168.1.25 - - [23/Apr/2020:16:15:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:59:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
现在需要编写Shell脚本统计每分钟的请求数，并且按照请求数降序排序。你的脚本应该输出：
5 20:27
4 15:00
2 22:10
2 14:12
2 10:27
1 23:59
1 21:21
1 16:15
1 15:26
1 09:20
1 08:05
1
2
3
4
5
6
7
8
9
10
#!/bin/bash

#1:
cat nowcoder.txt |awk -F':' '{print $2":"$3}' |sort |uniq -c |sort -r |awk '{print $1" "$2}'

#2:
awk '{arr[substr($4,14,5)]++}END{for(i in arr) print arr[i]" "i}' nowcoder.txt |sort -k 1 -r

#3:
awk -F ":" '{arr[($2":"$3)]++}END{for (i in arr) {print arr[i], i}}' | sort -h -r

29、netstat练习1-查看各个状态的连接数

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
假设netstat命令运行的结果我们存储在nowcoder.txt里，格式如下：
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:6160            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 172.16.56.200:41856     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49822     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49674     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:42316     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:44076     172.16.240.74:6379      ESTABLISHED
tcp        0      0 172.16.56.200:49656     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:58248     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:50108     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41944     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:35548     100.100.32.118:80       TIME_WAIT
tcp        0      0 172.16.56.200:39024     100.100.45.106:443      TIME_WAIT
tcp        0      0 172.16.56.200:41788     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58260     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:41812     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:41854     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58252     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:49586     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41754     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:50466     120.55.222.235:80       TIME_WAIT
tcp        0      0 172.16.56.200:38514     100.100.142.5:80        TIME_WAIT
tcp        0      0 172.16.56.200:49832     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:52162     100.100.30.25:80        ESTABLISHED
tcp        0      0 172.16.56.200:50372     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:50306     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49600     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41908     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:60292     100.100.142.1:80        TIME_WAIT
tcp        0      0 172.16.56.200:37650     100.100.54.133:80       TIME_WAIT
tcp        0      0 172.16.56.200:41938     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49736     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41890     172.16.34.144:3306      ESTABLISHED
udp        0      0 127.0.0.1:323           0.0.0.0:*
udp        0      0 0.0.0.0:45881           0.0.0.0:*
udp        0      0 127.0.0.53:53           0.0.0.0:*
udp        0      0 172.16.56.200:68        0.0.0.0:*
udp6       0      0 ::1:323                 :::*
raw6       0      0 :::58                   :::*                    7
现在需要编写shell脚本查看系统tcp连接中各个状态的连接数，并且按照连接数降序输出。你的脚本应该输出如下：
ESTABLISHED 22
TIME_WAIT 9
LISTEN 3
输入描述：
1
输出描述：
1
示例1
输入：
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:6160            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 172.16.56.200:41856     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49822     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49674     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:42316     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:44076     172.16.240.74:6379      ESTABLISHED
tcp        0      0 172.16.56.200:49656     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:58248     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:50108     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41944     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:35548     100.100.32.118:80       TIME_WAIT
tcp        0      0 172.16.56.200:39024     100.100.45.106:443      TIME_WAIT
tcp        0      0 172.16.56.200:41788     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58260     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:41812     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:41854     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58252     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:49586     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41754     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:50466     120.55.222.235:80       TIME_WAIT
tcp        0      0 172.16.56.200:38514     100.100.142.5:80        TIME_WAIT
tcp        0      0 172.16.56.200:49832     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:52162     100.100.30.25:80        ESTABLISHED
tcp        0      0 172.16.56.200:50372     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:50306     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49600     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41908     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:60292     100.100.142.1:80        TIME_WAIT
tcp        0      0 172.16.56.200:37650     100.100.54.133:80       TIME_WAIT
tcp        0      0 172.16.56.200:41938     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49736     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41890     172.16.34.144:3306      ESTABLISHED
udp        0      0 127.0.0.1:323           0.0.0.0:*
udp        0      0 0.0.0.0:45881           0.0.0.0:*
udp        0      0 127.0.0.53:53           0.0.0.0:*
udp        0      0 172.16.56.200:68        0.0.0.0:*
udp6       0      0 ::1:323                 :::*
raw6       0      0 :::58                   :::*                    7
复制
输出：
ESTABLISHED 22
TIME_WAIT 9
LISTEN 3
1
2
3
4
5
6
7
8
9
#awk数组+sort+正则：
awk '/^tcp/{
    a[$6]++;
}
END{
    for(i in a){
        print(i" "a[i])
    }
}' nowcoder.txt | sort -k2nr
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
#awk:
awk '{
    if ($1 == "tcp") {
        arr[$6]++
    }
} END {
    for (i in arr) {
        printf("%s %d\n", i, arr[i])
    }
}' | sort -nrk2

#shell:
declare -A map
while read line; do
	tmp=($line)
	[ ${tmp[0]} == "tcp" ] && ((map["${tmp[5]}"]++))
done <nowcoder.txt
function InsertSort() {
	tmp=()
	for ve in ${map[*]}; do
		tmp[${#tmp[*]}]=$ve
	done
	q=${#tmp[*]}
	for ((i = 0; i < $q; i++)); do
		for ((j = $i + 1; j < $q; j++)); do
			if [ ${tmp[$i]} -lt ${tmp[$j]} ]; then
				t=${tmp[$i]}
				tmp[$i]=${tmp[$j]}
				tmp[$j]=$t
			fi
		done
	done
}
InsertSort
for ((i = 0; i < q; i++)); do
	for ve in ${!map[*]}; do
		if [ ${tmp[$i]} -eq ${map[$ve]} ]; then
			printf "$ve ${map[$ve]}\n"
		fi
	done
done

30、netstat练习2-查看和3306端口建立的连接

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
假设netstat命令运行的结果我们存储在nowcoder.txt里，格式如下：
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:6160            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 172.16.56.200:41856     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49822     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49674     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:42316     172.16.34.143:3306      ESTABLISHED
tcp        0      0 172.16.56.200:44076     172.16.240.74:6379      ESTABLISHED
tcp        0      0 172.16.56.200:49656     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:58248     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:50108     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41944     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:35548     100.100.32.118:80       TIME_WAIT
tcp        0      0 172.16.56.200:39024     100.100.45.106:443      TIME_WAIT
tcp        0      0 172.16.56.200:41788     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58260     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:41812     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:41854     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58252     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:49586     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41754     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:50466     120.55.222.235:80       TIME_WAIT
tcp        0      0 172.16.56.200:38514     100.100.142.5:80        TIME_WAIT
tcp        0      0 172.16.56.200:49832     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:52162     100.100.30.25:80        ESTABLISHED
tcp        0      0 172.16.56.200:50372     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:50306     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49600     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41908     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:60292     100.100.142.1:80        TIME_WAIT
tcp        0      0 172.16.56.200:37650     100.100.54.133:80       TIME_WAIT
tcp        0      0 172.16.56.200:41938     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49736     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41890     172.16.34.144:3306      ESTABLISHED
udp        0      0 127.0.0.1:323           0.0.0.0:*
udp        0      0 0.0.0.0:45881           0.0.0.0:*
udp        0      0 127.0.0.53:53           0.0.0.0:*
udp        0      0 172.16.56.200:68        0.0.0.0:*
udp6       0      0 ::1:323                 :::*
raw6       0      0 :::58                   :::*                    7
现在需要你查看和本机3306端口建立连接并且状态是established的所有IP，按照连接数降序排序。你的脚本应该输出
10 172.16.0.24
9 172.16.34.144
1 172.16.34.143
1
2
3
4
5
6
7
8
9
10
11
12
#awk+gensub+sort:
awk '{
    if($6 == "ESTABLISHED" && $5 ~ /.*:3306/){
        str=gensub(/(.*):3306/, "\\1", "g", $5)
        a[str]++
    }
}
END{
    for(i in a){
        print(a[i]" "i)
    }
}' nowcoder.txt | sort -nr
1
2
3
4
5
#cut 过滤出链接的ip
#使用ip地址作为下标，然后进行统计
#最后使用sort进行排序

cat nowcoder.txt|grep 3306|grep ESTABLISHED|cut -d : -f 2|awk '{a[$2]++};END{for (i in a)print a[i],i}'|sort -nr
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
#希尔排序:
declare -A map
while read line
    do
        tmp=($line)
        ip=(${tmp[4]/:/ })
        [ ${tmp[0]} == "tcp" ] && [ ${tmp[5]} == "ESTABLISHED" ] && [ ${ip[1]} -eq 3306 ] &&  ((map["${ip[0]}"]++))
    done < nowcoder.txt
    tmp=()
    for ve in ${map[*]}
        do
            tmp[${#tmp[*]}]=$ve
        done
    q=${#tmp[*]}
for (( gap=q>>1; gap>0; gap=gap>>1 ))
    do
        for (( i=gap; i<q; i++ ))
            do
                temp=${tmp[$i]}
                for (( a=i-gap; a>=0&temp>${tmp[$a]}; a-=gap ))  #默认升序, 如果降序排序改为(( a=i-gap; a>=0&temp>${arr[$a]}; a-=gap ))
                    do
                        tmp[$a+$gap]=${tmp[$a]}
                    done
                tmp[$a+$gap]=$temp
            done
    done
for ((i=0; i<$q; i++))
    do
        for ve in ${!map[*]}
            do
                if [ ${tmp[$i]} -eq ${map[$ve]} ];then
                    printf "${map[$ve]} $ve\n"
                fi
            done
    done

31、netstat练习3-输出每个IP的连接数

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
假设netstat命令运行的结果我们存储在nowcoder.txt里，格式如下：
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:6160            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 172.16.56.200:41856     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49822     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49674     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:42316     172.16.34.143:3306      ESTABLISHED
tcp        0      0 172.16.56.200:44076     172.16.240.74:6379      ESTABLISHED
tcp        0      0 172.16.56.200:49656     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:58248     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:50108     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41944     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:35548     100.100.32.118:80       TIME_WAIT
tcp        0      0 172.16.56.200:39024     100.100.45.106:443      TIME_WAIT
tcp        0      0 172.16.56.200:41788     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58260     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:41812     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:41854     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58252     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:49586     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41754     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:50466     120.55.222.235:80       TIME_WAIT
tcp        0      0 172.16.56.200:38514     100.100.142.5:80        TIME_WAIT
tcp        0      0 172.16.56.200:49832     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:52162     100.100.30.25:80        ESTABLISHED
tcp        0      0 172.16.56.200:50372     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:50306     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49600     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41908     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:60292     100.100.142.1:80        TIME_WAIT
tcp        0      0 172.16.56.200:37650     100.100.54.133:80       TIME_WAIT
tcp        0      0 172.16.56.200:41938     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49736     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41890     172.16.34.144:3306      ESTABLISHED
udp        0      0 127.0.0.1:323           0.0.0.0:*
udp        0      0 0.0.0.0:45881           0.0.0.0:*
udp        0      0 127.0.0.53:53           0.0.0.0:*
udp        0      0 172.16.56.200:68        0.0.0.0:*
udp6       0      0 ::1:323                 :::*
raw6       0      0 :::58                   :::*                    7
现在需要你输出每个IP的连接数，按照连接数降序排序。你的脚本应该输出
172.16.0.24 10
172.16.34.144 9
100.100.142.4 3
0.0.0.0 3
172.16.34.143 1
172.16.240.74 1
120.55.222.235 1
100.100.54.133 1
100.100.45.106 1
100.100.32.118 1
100.100.30.25 1
100.100.142.5 1
100.100.142.1 1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
#1：
grep tcp nowcoder.txt | awk '{print $5}' | awk -F : '{print $1}' | sort | uniq -c | sort -rn | awk '{print $2,$1}'
#2：
awk -F '[ :]+' '{if($8 ~ /[A-Z\_]+/ && $6 ~ /[0-9\.]+/ ){print $6}}' nowcoder.txt | sort | uniq -c | awk '{print $2" "$1}'| sort -rnk2
#3：
awk -F '[ :]+' 'BEGIN{}{
    if($8 ~ /[A-Z\_]+/ && $6 ~ /[0-9\.]+/){
        dict[$6]++
    }
}END{
    for(each_ip in dict){
        printf("%s %d\n", each_ip, dict[each_ip])
    }
}' nowcoder.txt | sort -rnk2

32、netstat练习4-输出和3306端口建立连接总的各个状态的数目

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
假设netstat命令运行的结果我们存储在nowcoder.txt里，格式如下：
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:6160            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 172.16.56.200:41856     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49822     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49674     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:42316     172.16.34.143:3306      ESTABLISHED
tcp        0      0 172.16.56.200:44076     172.16.240.74:6379      ESTABLISHED
tcp        0      0 172.16.56.200:49656     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:58248     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:50108     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41944     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:35548     100.100.32.118:80       TIME_WAIT
tcp        0      0 172.16.56.200:39024     100.100.45.106:443      TIME_WAIT
tcp        0      0 172.16.56.200:41788     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58260     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:41812     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:41854     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58252     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:49586     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41754     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:50466     120.55.222.235:80       TIME_WAIT
tcp        0      0 172.16.56.200:38514     100.100.142.5:80        TIME_WAIT
tcp        0      0 172.16.56.200:49832     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:52162     100.100.30.25:80        ESTABLISHED
tcp        0      0 172.16.56.200:50372     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:50306     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49600     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41908     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:60292     100.100.142.1:80        TIME_WAIT
tcp        0      0 172.16.56.200:37650     100.100.54.133:80       TIME_WAIT
tcp        0      0 172.16.56.200:41938     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49736     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41890     172.16.34.144:3306      ESTABLISHED
udp        0      0 127.0.0.1:323           0.0.0.0:*
udp        0      0 0.0.0.0:45881           0.0.0.0:*
udp        0      0 127.0.0.53:53           0.0.0.0:*
udp        0      0 172.16.56.200:68        0.0.0.0:*
udp6       0      0 ::1:323                 :::*
raw6       0      0 :::58                   :::*                    7
现在需要你输出和本机3306端口建立连接的各个状态的数目，按照以下格式输出
TOTAL_IP 3
ESTABLISHED 20
TOTAL_LINK 20
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#awk+gensub:

awk '{    
    if($5 ~ /.*:3306/){
        str=gensub(/(.*):3306/, "\\1", "g", $5)
        if(!(str in a)) {
            ++total_ip
            a[str]++
        }
        if($6 == "ESTABLISHED"){
            ++established
        }
        ++total_link
    }
}
END{
    print("TOTAL_IP "total_ip"\n""ESTABLISHED "established"\n""TOTAL_LINK "total_link)
}' nowcoder.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/bin/bash

# 业务知识
# 需要了解netstat命令运行的结果

# 问题提炼：本机3306端口   建立连接(ESTABLISHED)    各个状态的数目

# 思路：结果导向
# awk  $5 ~ /3306/
# 汇总不同的IP   关联数组++时 等于1  total_ip++
# 汇总 $3 ~ /ESTABLISHED/
# 汇总 IP

awk '
    {
        if ($5 ~ /3306/)
            print $5":"$6
    }
' nowcoder.txt |
# 172.16.0.24:3306:ESTABLISHED
awk -F ":" '
    BEGIN{
        total_ip = 0
        es = 0
        total_link = 0
    }
    {
        if (++arr[$1] == 1)
            total_ip++
        if ($3 ~ /ESTABLISHED/)
            es++     
    }
    END{
        print "TOTAL_IP "    total_ip
        print "ESTABLISHED " es
        for (i in arr)
            total_link += arr[i]
        print "TOTAL_LINK " total_link
    }
'

33、业务分析-提取值

描述

1
2
3
4
5
6
7
8
9
10
11
假设我们的日志nowcoder.txt里，内容如下
12-May-2017 10:02:22.789 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:Apache Tomcat/8.5.15
12-May-2017 10:02:22.813 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:May 5 2017 11:03:04 UTC
12-May-2017 10:02:22.813 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log Server number:8.5.15.0
12-May-2017 10:02:22.814 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:Windows, OS Version:10
12-May-2017 10:02:22.814 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:x86_64
现在需要你提取出对应的值，输出内容如下
serverVersion:Apache Tomcat/8.5.15
serverName:8.5.15.0
osName:Windows
osVersion:10
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#awk+gensub+正则:
awk '{
    if($6 == "Server") {
        if($7 ~/version:.*/){
            sversion = gensub(/(.*)version:(.*)/, "\\2", "g", $0) 
        }
        else if($7 ~/number:.*/){
            sname = gensub(/(.*)number:(.*)/, "\\2", "g", $0) 
        }
    }
    else if($6 == "OS"){
        oname = gensub(/(.*)Name:(.*),(.*)/, "\\2", "g", $0)
        oversion = gensub(/(.*)Version:(.*)/, "\\2", "g", $0)
    }
}
END{
    print("serverVersion:"sversion"\n""serverName:"sname"\n""osName:"oname"\n""osVersion:"oversion)
}' nowcoder.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#!/bin/bash

#数据保存
declare -A targetArr=(["Server_version"]='' ['Server_number']='' ["OS_Name"]='' ['OS_Version']='')
#输出转换
declare -A translateArr=(['Server_version']='serverVersion' ['Server_number']='serverName' ['OS_Name']='osName' ['OS_Version']='osVersion')

while read line; do
	{
		for key in ${!targetArr[@]}; do {
			str=${targetArr[$key]}
			#存在值则不操作
			if ((${#str} == 0)); then {
				match=${key/_/ }
				if [[ $line =~ ($match)(:)([^,]+) ]]; then
					#提取第三段value
					targetArr[$key]=${BASH_REMATCH[3]}
				fi
			}; fi
		}; done
	}
done
for key in ${!targetArr[@]}; do {
	#通过翻译数组输出
	echo ${translateArr[$key]}:${targetArr[$key]}
}; done

34、ps分析-统计VSZ,RSS各自总和

描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
假设命令运行的结果我们存储在nowcoder.txt里，格式如下：
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  37344  4604 ?        Ss    2020   2:13 /sbin/init
root       231  0.0  1.5 166576 62740 ?        Ss    2020  15:15 /lib/systemd/systemd-journald
root       237  0.0  0.0      0     0 ?        S<    2020   2:06 [kworker/0:1H]
root       259  0.0  0.0  45004  3416 ?        Ss    2020   0:25 /lib/systemd/systemd-udevd
root       476  0.0  0.0      0     0 ?        S<    2020   0:00 [edac-poller]
root       588  0.0  0.0 276244  2072 ?        Ssl   2020   9:49 /usr/lib/accountsservice/accounts-daemon
message+   592  0.0  0.0  42904  3032 ?        Ss    2020   0:01 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root       636  0.0  0.0  65532  3200 ?        Ss    2020   1:51 /usr/sbin/sshd -D
daemon     637  0.0  0.0  26044  2076 ?        Ss    2020   0:00 /usr/sbin/atd -f
root       639  0.0  0.0  29476  2696 ?        Ss    2020   3:29 /usr/sbin/cron -f
root       643  0.0  0.0  20748  1992 ?        Ss    2020   0:26 /lib/systemd/systemd-logind
syslog     645  0.0  0.0 260636  3024 ?        Ssl   2020   3:17 /usr/sbin/rsyslogd -n
root       686  0.0  0.0 773124  2836 ?        Ssl   2020  26:45 /usr/sbin/nscd
root       690  0.0  0.0  19472   252 ?        Ss    2020  14:39 /usr/sbin/irqbalance --pid=/var/run/irqbalance.pid
ntp        692  0.0  0.0  98204   776 ?        Ss    2020  25:18 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 108:114
uuidd      767  0.0  0.0  28624   192 ?        Ss    2020   0:00 /usr/sbin/uuidd --socket-activation
root       793  0.0  0.0 128812  3148 ?        Ss    2020   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data   794  0.0  0.2 133376  9120 ?        S     2020 630:57 nginx: worker process
www-data   795  0.0  0.2 133208  8968 ?        S     2020 633:02 nginx: worker process
www-data   796  0.0  0.2 133216  9120 ?        S     2020 634:24 nginx: worker process
www-data   797  0.0  0.2 133228  9148 ?        S     2020 632:56 nginx: worker process
web        955  0.0  0.0  36856  2112 ?        Ss    2020   0:00 /lib/systemd/systemd --user
web        956  0.0  0.0  67456  1684 ?        S     2020   0:00 (sd-pam)
root      1354  0.0  0.0   8172   440 tty1     Ss+   2020   0:00 /sbin/agetty --noclear tty1 linux
root      1355  0.0  0.0   7988   344 ttyS0    Ss+   2020   0:00 /sbin/agetty --keep-baud 115200 38400 9600 ttyS0 vt220
root      2513  0.0  0.0      0     0 ?        S    13:07   0:00 [kworker/u4:1]
root      2587  0.0  0.0      0     0 ?        S    13:13   0:00 [kworker/u4:2]
root      2642  0.0  0.0      0     0 ?        S    13:17   0:00 [kworker/1:0]
root      2679  0.0  0.0      0     0 ?        S    13:19   0:00 [kworker/u4:0]
root      2735  0.0  0.1 102256  7252 ?        Ss   13:24   0:00 sshd: web [priv]
web       2752  0.0  0.0 102256  3452 ?        R    13:24   0:00 sshd: web@pts/0
web       2753  0.5  0.1  14716  4708 pts/0    Ss   13:24   0:00 -bash
web       2767  0.0  0.0  29596  1456 pts/0    R+   13:24   0:00 ps aux
root     10634  0.0  0.0      0     0 ?        S    Nov16   0:00 [kworker/0:0]
root     16585  0.0  0.0      0     0 ?        S<    2020   0:00 [bioset]
root     19526  0.0  0.0      0     0 ?        S    Nov16   0:00 [kworker/1:1]
root     28460  0.0  0.0      0     0 ?        S    Nov15   0:03 [kworker/0:2]
root     30685  0.0  0.0  36644  2760 ?        Ss    2020   0:00 /lib/systemd/systemd --user
root     30692  0.0  0.0  67224  1664 ?        S     2020   0:00 (sd-pam)
root     32689  0.0  0.0  47740  2100 ?        Ss    2020   0:00 /usr/local/ilogtail/ilogtail
root     32691  0.2  0.5 256144 23708 ?        Sl    2020 1151:31 /usr/local/ilogtail/ilogtail
现在需要你统计VSZ，RSS各自的总和（以M兆为统计），输出格式如下
MEM TOTAL
VSZ_SUM:3250.8M,RSS_SUM:179.777M
1
2
3
4
5
6
7
8
9
10
11
#纯awk:
awk '{
    v += $5
    r += $6
} END {
    printf("MEM TOTAL\nVSZ_SUM:%0.1fM,RSS_SUM:%0.3fM", v/1024, r/1024)
}'

#请问这个直接加，是会自动忽略第一行非数字的名称内容吗:是的，因为从第二行开始赋值数字 会把第一行的非数字的字符串给替换掉成为int类型
#请问为什么会自动忽略第一行呢，还以为要加个判断:这个和awk的匹配模式有关，因为+=操作是对数字进行的，awk是分别读入每一行数据，如果条件不匹配，就继续下一行，字符就被跳过了。
加判断条件也可以，不过也可以加一个sed '1d' nowcoder.txt
1
2
3
4
5
6
7
8
9
#注意精度以及单位即可，NR>1也可以不加，因为awk会在进行数值运算时将字符串转成0
awk 'NR > 1{
    vsz_sum+=$5
    rss_sum+=$6
}
END{
    print("MEM TOTAL")
    printf("VSZ_SUM:%.1fM,RSS_SUM:%.3fM", vsz_sum / 2^10, rss_sum / 2^10)
}' nowcoder.txt