Automação de teste do sistema baseado em QEMU (Parte 2/2)

Esta é a segunda parte de um artigo que cobre a automação de teste do sistema baseado em máquina virtual. A primeira parte pode ser encontrada aqui .



Nesta parte do artigo, usaremos as habilidades aprendidas na primeira parte para realmente automatizar os testes do sistema. No final do artigo, receberemos um script que qualquer pessoa pode executar em seu computador e, do zero, obter um stand implantado de três máquinas, instalado pelo aplicativo em teste, bem como passar em testes de sistema real (escreveremos três testes).



aviso Legal

, , , . " " , . , .



Portanto, na última parte, acumulamos um arsenal impressionante de habilidades para trabalhar com máquinas virtuais na linha de comando: aprendemos como instalar máquinas virtuais, implantar um sistema operacional nelas (por exemplo, Ubuntu Server 18.04), conectar uma máquina virtual a um host em uma rede e até mesmo organizar um canal de controle via SSH. Tudo isso será útil para nós neste artigo, mas antes de prosseguirmos para a prática, há vários assuntos a serem discutidos.



O que queremos obter?



A pergunta mais importante a ser respondida é "Que resultado queremos obter?" Sim, da última vez falamos muito sobre automatizar a instalação, implantação e configuração de máquinas virtuais, mas fora o objetivo final, tudo isso não faz muito sentido.



Para mim, pessoalmente, os testes de sistema tudo-em-um têm a seguinte aparência: eu faço download de vários pequenos arquivos do VCS (o próprio script com o lançamento, além de, possivelmente, vários artefatos auxiliares), coloco o programa em teste onde eu preciso (na forma de um instalador ou um pacote, por exemplo), Aperto um botão e vou tomar um café. Quando eu voltar, quero ver se todos os testes foram aprovados ou se tais e tais testes foram interrompidos. Não quero me envolver em nenhuma configuração do estande, não quero implantar máquinas virtuais ou montar algo lá. Quero baixar um script e usá-lo.



, : , . , ( ).



, .





. -, Data Plane Development Kit (DPDK). DPDK — , , , DPDK. DPDK , , end-to-end .



DPDK

DPDK (Data Plane Development Kit) — , C. , . , . , . DPDK . ? . , , Linux, , , , . , Linux — , . , DPDK .



:



  1. ;
  2. ;
  3. DROP — ;
  4. ACCEPT — ;


, , :



  • ;
  • , ;




, , , :



  1. (client, middlebox, server), Ubuntu Server 18.04 ();
  2. : client middlebox ( net_1) middlebox server ( net_2);
  3. ;
  4. , ;
  5. - middlebox;
  6. .


. :



  1. , SSH- , , , ;
  2. SSH net_1 net_2, ( net_for_ssh). :


  • .. , , - ;
  • , - (, ), .


!



- :







, , :



run_tests.sh
#!/bin/bash
set -euo pipefail

# =======================================
#   net_for_ssh
# =======================================
virsh net-define net_for_ssh.xml
virsh net-start net_for_ssh

# =======================================
#   net_1
# =======================================
virsh net-define net_1.xml
virsh net-start net_1

# =======================================
#   net_2
# =======================================
virsh net-define net_2.xml
virsh net-start net_2

# =======================================
#   client
# =======================================
virt-builder ubuntu-18.04 \
    --format qcow2 \
    --output client.qcow2 \
    --install wget \
    --root-password password:1111 \
    --run-command "ssh-keygen -A" \
    --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
    --copy-in netcfg_client.yaml:/etc/netplan/

virt-install \
    --import \
    --name client \
    --ram 1024 \
    --disk client.qcow2 \
    --network network=net_for_ssh \
    --network network=net_1,mac=52:54:56:11:00:00 \
    --noautoconsole

# =======================================
#   middlebox
# =======================================
virt-builder ubuntu-18.04 \
    --format qcow2 \
    --output middlebox.qcow2 \
    --install python,daemon,libnuma1 \
    --root-password password:1111 \
    --run-command "ssh-keygen -A" \
    --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
    --copy-in netcfg_middlebox.yaml:/etc/netplan/

virt-install \
    --import \
    --name middlebox \
    --vcpus=2,sockets=1,cores=2,threads=1 \
    --cpu host \
    --ram 2048 \
    --disk middlebox.qcow2 \
    --network network=net_for_ssh \
    --network network=net_1,model=e1000 \
    --network network=net_2,model=e1000 \
    --noautoconsole

# =======================================
#   server
# =======================================
virt-builder ubuntu-18.04 \
    --format qcow2 \
    --output server.qcow2 \
    --install nginx \
    --root-password password:1111 \
    --run-command "ssh-keygen -A" \
    --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
    --copy-in netcfg_server.yaml:/etc/netplan/

virt-install \
    --import \
    --name server \
    --ram 1024 \
    --disk server.qcow2 \
    --network network=net_for_ssh \
    --network network=net_2,mac=52:54:56:00:00:00 \
    --noautoconsole

# =======================================
# ,    
#     
# =======================================

SSH_CMD="sshpass -p 1111 ssh -o StrictHostKeyChecking=no"

while ! SSH_CMD root@192.168.100.2 "echo Hello world from client!" echo
do
    echo "Waiting for client VM ..."
    sleep 1
done

while ! SSH_CMD root@192.168.100.3 "echo Hello world from middlebox!" echo
do
    echo "Waiting for middlebox VM ..."
    sleep 1
done

while ! SSH_CMD root@192.168.100.4 "echo Hello world from server!" echo
do
    echo "Waiting for server VM ..."
    sleep 1
done


:



net_for_ssh.xml
<network>
    <name>net_for_ssh</name>
    <bridge name='net_for_ssh'/>
    <ip address='192.168.100.1' netmask='255.255.255.0'/>
</network>


net_1.xml
<network>
    <name>net_1</name>
    <bridge name='net_1'/>
    <ip address='192.168.101.1' netmask='255.255.255.0'/>
</network>


net_2.xml
<network>
    <name>net_2</name>
    <bridge name='net_2'/>
    <ip address='192.168.102.1' netmask='255.255.255.0'/>
</network>


netcfg_client.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.100.2/24
    ens4:
      addresses:
        - 192.168.101.2/24
      gateway4: 192.168.101.3


netcfg_middlebox.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.100.3/24


netcfg_server.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.100.4/24
    ens4:
      addresses:
        - 192.168.102.2/24
      gateway4: 192.168.102.3


, :



  1. --install virt-builder, . --run-command "apt install ...". , , — virt-builder . client wget, servernginx ( http- ). middlebox , DPDK-;
  2. client server , - , . ;
  3. middlebox ( --vcpus): CPU c hyperthreading. — , DPDK . --cpu host, , , . , - QEMU , SSE3 . , , DPDK .
  4. middlebox , : e1000. , - DPDK.


run_tests.sh ( , ). , :



  1. ;
  2. .


, run_tests.sh, , . :



run_clean.sh
#!/bin/bash
set -euo pipefail

# =======================================
#   client
# =======================================
if virsh list --all | grep -q " client "; then
    if virsh domstate client | grep -q "running"; then
        virsh destroy client
    fi
    virsh undefine client --snapshots-metadata --remove-all-storage
fi

# =======================================
#   middlebox
# =======================================
if virsh list --all | grep -q " middlebox "; then
    if virsh domstate middlebox | grep -q "running"; then
        virsh destroy middlebox
    fi
    virsh undefine middlebox --snapshots-metadata --remove-all-storage
fi

# =======================================
#   server
# =======================================
if virsh list --all | grep -q " server "; then
    if virsh domstate server | grep -q "running"; then
        virsh destroy server
    fi
    virsh undefine server --snapshots-metadata --remove-all-storage
fi

# =======================================
#   net_for_ssh
# =======================================
if virsh net-list --all | grep -q " net_for_ssh "; then
    if virsh net-list --all | grep " net_for_ssh " | grep -q " active "; then
        virsh net-destroy net_for_ssh
    fi
    virsh net-undefine net_for_ssh
fi

# =======================================
#   net_1
# =======================================
if virsh net-list --all | grep -q " net_1 "; then
    if virsh net-list --all | grep " net_1 " | grep -q " active "; then
        virsh net-destroy net_1
    fi
    virsh net-undefine net_1
fi

# =======================================
#   net_2
# =======================================
if virsh net-list --all | grep -q " net_2 "; then
    if virsh net-list --all | grep " net_2 " | grep -q " active "; then
        virsh net-destroy net_2
    fi
    virsh net-undefine net_2
fi


, run_tests.sh, . :



  1. ( virsh list --all, — virsh net-list -all);
  2. /, , / , ;
  3. ( --snapshots-metadata) ( --remove-all-storage).


run_tests.sh run_clean.sh. run_tests.sh , run_clean.sh.



run_clean.sh . . , , , .



. SSH- — SSH ~/.ssh/known_hosts. , SSH — , , - . SSH_CMD:



SSH_CMD="sshpass -p 1111 ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR"


-o UserKnownHostsFile=/dev/null, .





, — . Ubuntu Server ( GUI) bash-, .



bash-, . , .



-, , :



EXEC_CLIENT="$SSH_CMD root@192.168.100.2"
EXEC_MIDDLEBOX="$SSH_CMD root@192.168.100.3"
EXEC_SERVER="$SSH_CMD root@192.168.100.4"


. . :



$EXEC_CLIENT echo hello from client
$EXEC_SERVER echo hello from server


-, ? bash- Heredoc. Heredoc :



$EXEC_CLIENT << EOF
    echo hello from client
    ls
    pwd
EOF

$EXEC_MIDDLEBOX << EOF
    echo hello from middlebox
    some_another_command
EOF


EOF — , . EOF , , .



, set -xeuo pipefail. :



$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    command1
    command2 | command3
EOF


, :



  • -x bash- ;
  • -e , ;
  • -u , ;
  • -o pipeline , - .


, - — .



, . bash- , - . :



$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    command1
    ! command2
EOF


: command1 0, command2 , .





:



  1. , ;
  2. , ;
  3. , .


:



$SCP_CMD l3fwd-acl-1.0.0.deb root@192.168.100.3:~

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    dpkg -i l3fwd-acl-1.0.0.deb

    echo 256 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
    mkdir -p /mnt/huge
    mount -t hugetlbfs nodev /mnt/huge

    modprobe uio_pci_generic
    dpdk-devbind --bind=uio_pci_generic ens4 ens5

    echo "R0.0.0.0/0 192.168.102.0/24 0 : 65535 0 : 65535 0x0/0x0 1" > /etc/rule_ipv4.db
    echo "R0.0.0.0/0 192.168.101.0/24 0 : 65535 0 : 65535 0x0/0x0 0" >> /etc/rule_ipv4.db

    echo "R0:0:0:0:0:0:0:0/0 0:0:0:0:0:0:0:0/0 0 : 65535 0 : 65535 0x0/0x0 0" > /etc/rule_ipv6.db

    daemon --name l3fwd --unsafe --output /var/log/l3fwd -- l3fwd-acl \
        -l 1 \
        -n 4 \
        -- \
        -p 0x3 \
        -P \
        --config="(0,0,1),(1,0,1)" \
        --rule_ipv4="/etc/rule_ipv4.db" \
        --rule_ipv6="/etc/rule_ipv6.db"
EOF


, ( ):



  1. deb-;
  2. 256 2 (DPDK- - );
  3. poll-mode uio_pci_generic ( Ubuntu Server). , DPDK ;
  4. ens4 ( ) ens5 ( ) uio_pci_generic;
  5. rule_ipv4.db IPv4 : 192.168.102.0/24 1 ( , ), 192.168.101.0/24 0 ( );
  6. rule_ipv6.db, " 0". IPv6 , DPDK ;
  7. l3fwd daemon. , l3fwd: https://doc.dpdk.org/guides/sample_app_ug/l3_forward.html


, DPDK :



  1. .deb ;
  2. ;
  3. ;
  4. ;
  5. , , uio_pci_generic;
  6. .


: " ", - . , . .



: , DPDK- ( unit-, ), , : - :



$EXEC_CLIENT arp -s 192.168.101.3 52:54:56:00:00:00
$EXEC_SERVER arp -s 192.168.102.3 52:54:56:11:00:00

$EXEC_CLIENT << EOF
    set -xeuo pipefail

    ping -c 5 192.168.102.2
    wget --timeout=5 --tries=1 http://192.168.102.2
EOF


ARP-

, , ARP-.

? , l3fwd,

DPDK, , ARP.

l3fwd ,

rule_ipv4.db rule_ipv6.db,

: , / , --.

:

, .

, middlebox

, MAC-

Ethernet- ( client server ).

:

destination MAC-.

ARP-.



:



# =======================================
#  ,  tcp 
# =======================================

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    daemon --name l3fwd --stop

    #     
    echo "@0.0.0.0/0 0.0.0.0/0 0 : 65535 0 : 65535 0x06/0xff" > /etc/rule_ipv4.db

    echo "R0.0.0.0/0 192.168.102.0/24 0 : 65535 0 : 65535 0x0/0x0 1" >> /etc/rule_ipv4.db
    echo "R0.0.0.0/0 192.168.101.0/24 0 : 65535 0 : 65535 0x0/0x0 0" >> /etc/rule_ipv4.db

    daemon --name l3fwd --unsafe --output /var/log/l3fwd -- l3fwd-acl \
        -l 1 \
        -n 4 \
        -- \
        -p 0x3 \
        -P \
        --config="(0,0,1),(1,0,1)" \
        --rule_ipv4="/etc/rule_ipv4.db" \
        --rule_ipv6="/etc/rule_ipv6.db"
EOF

# =======================================
# ,  ping  ,
#  http  - 
# =======================================

$EXEC_CLIENT << EOF
    set -xeuo pipefail

    ping -c 5 192.168.102.2
    ! wget --timeout=5 --tries=1 http://192.168.102.2
EOF


, wget : , , wget .



run_tests.sh



, , run_clean.sh.



: : DPDK- middlebox . ? , ( ) DPDK- . , , . ?



. , ? , . : , init, .



:



# =======================================
#   client
# =======================================
if ! virsh list --all | grep -q " client "
then
    virt-builder ubuntu-18.04 \
        --format qcow2 \
        --output client.qcow2 \
        --hostname client \
        --install wget,net-tools \
        --root-password password:1111 \
        --run-command "ssh-keygen -A" \
        --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
        --copy-in netcfg_client.yaml:/etc/netplan/

    virt-install \
        --import \
        --name client \
        --ram 1024 \
        --disk client.qcow2 \
        --network network=net_for_ssh \
        --network network=net_1,mac=52:54:56:11:00:00 \
        --noautoconsole

    virsh snapshot-create-as client --name init
else
    virsh snapshot-revert client --snapshotname init
fi


, init.



, , : , .deb- DPDK, . .



, , :



# =======================================
#   net_1
# =======================================
if ! virsh net-list --all | grep -q " net_1 "
then
    virsh net-define net_1.xml
    virsh net-start net_1
fi


, , , . : !





, , , , . : , , , (end-to-end) .



, , : DPDK- ( Ubuntu Server 18.04) ( ping wget). , .deb . , , ( , ). .



: (run_tests.sh run_clean.sh), xml- yaml-. VCS. .



, , " — ". , . , " ", .



"", "" "" , , , , . Proof Of Concept . - … . , , , , . ...






All Articles