2019 November 05

libvirt代码调用流程分析

对于刚接触libvirt的新手来说，不清楚libvirt的体系架构，总是搞不清代码的调用流程，分不清客户端以及服务端。此篇文章有助于从简单的一个virsh命令出发，粗浅解析代码调用过程。

实验环境

宿主机系统（host）：CentOS Linux release 7.4.1708 (Core)

libvirt：3.2.0

qemu: 2.9.0

客户端与服务端

libvirt采用C/S模式，client端发送消息，server处理消息返回给client。virsh命令行相当于client，配置文件：

cat /etc/libvirt/libvirt.conf 
#
# This can be used to setup URI aliases for frequently
# used connection URIs. Aliases may contain only the
# characters  a-Z, 0-9, _, -.
#
# Following the '=' may be any valid libvirt connection
# URI, including arbitrary parameters

#uri_aliases = [
#  "hail=qemu+ssh://root@hail.cloud.example.com/system",
#  "sleet=qemu+ssh://root@sleet.cloud.example.com/system",
#]

#
# These can be used in cases when no URI is supplied by the application
# (@uri_default also prevents probing of the hypervisor driver).
#
#uri_default = "qemu:///system"
#uri_default = "qemu+tcp:///system

client端的方式还有很多种，官网给出了不同语言的client端，例如libvirt-python，libvirt-php等，方便用户调用libvirt接口。

服务端是libvirtd启动进程，接收客户端消息，配置文件为：

cat /etc/libvirt/libvirtd.conf 
# Master libvirt daemon configuration file
#
# For further information consult http://libvirt.org/format.html
#
# NOTE: the tests/daemon-conf regression test script requires
# that each "PARAMETER = VALUE" line in this file have the parameter
# name just after a leading "#".
...
###################################################################
# Open vSwitch:
# This allows to specify a timeout for openvswitch calls made by
# libvirt. The ovs-vsctl utility is used for the configuration and
# its timeout option is set by default to 5 seconds to avoid
# potential infinite waits blocking libvirt.
#
#ovs_timeout = 5

virsh端调用

以virsh resume虚拟机为例，来说明下代码调用:

(gdb) bt
#0  virNetClientSendWithReply (client=client@entry=0x55ab6653f300, msg=msg@entry=0x55ab6653e760) at rpc/virnetclient.c:2141
#1  0x00007f17d3da0a42 in virNetClientProgramCall (prog=prog@entry=0x55ab6653f530, client=client@entry=0x55ab6653f300, serial=serial@entry=13, proc=proc@entry=28, noutfds=noutfds@entry=0, 
    outfds=outfds@entry=0x0, ninfds=ninfds@entry=0x0, infds=infds@entry=0x0, args_filter=args_filter@entry=0x7f17d3d94ae0 <xdr_remote_domain_resume_args>, args=args@entry=0x7ffec7a7a780, 
    ret_filter=ret_filter@entry=0x7f17d0bce820 <xdr_void>, ret=ret@entry=0x0) at rpc/virnetclientprogram.c:329
#2  0x00007f17d3d74572 in callFull (priv=priv@entry=0x55ab6653e650, flags=flags@entry=0, fdin=fdin@entry=0x0, fdinlen=fdinlen@entry=0, fdout=fdout@entry=0x0, fdoutlen=fdoutlen@entry=0x0, 
    proc_nr=proc_nr@entry=28, args_filter=0x7f17d3d94ae0 <xdr_remote_domain_resume_args>, args=args@entry=0x7ffec7a7a780 "PEPf\253U", ret_filter=0x7f17d0bce820 <xdr_void>, 
    ret=ret@entry=0x0, conn=<optimized out>) at remote/remote_driver.c:6640
#3  0x00007f17d3d7a034 in call (conn=<optimized out>, ret=0x0, ret_filter=<optimized out>, args=0x7ffec7a7a780 "PEPf\253U", args_filter=<optimized out>, proc_nr=28, flags=0, 
    priv=0x55ab6653e650) at remote/remote_driver.c:6662
#4  remoteDomainResume (dom=0x55ab6653ed30) at remote/remote_client_bodies.h:3802
#5  0x00007f17d3d28abc in virDomainResume (domain=domain@entry=0x55ab6653ed30) at libvirt-domain.c:679
#6  0x000055ab660152ca in cmdResume (ctl=0x7ffec7a7a970, cmd=<optimized out>) at virsh-domain.c:5511
#7  0x000055ab66041bb0 in vshCommandRun (ctl=ctl@entry=0x7ffec7a7a970, cmd=0x55ab6653e8e0) at vsh.c:1312
#8  0x000055ab6600a3ba in main (argc=1, argv=<optimized out>) at virsh.c:988

具体过程：

virsh-domain.c
   {.name = "resume",
     .handler = cmdResume,
     .opts = opts_resume,
     .info = info_resume,
     .flags = 0
    },
hander：具体的处理函数
opts：命令行参数
info：命令相关信息，help resume可以看到info信息

继续看

libvirt-domain.c
int
virDomainResume(virDomainPtr domain)
{
    virConnectPtr conn;

    VIR_DOMAIN_DEBUG(domain);

    virResetLastError();

    virCheckDomainReturn(domain, -1);
    conn = domain->conn;

    virCheckReadOnlyGoto(conn->flags, error);

    if (conn->driver->domainResume) {
        int ret;
        ret = conn->driver->domainResume(domain);
        if (ret < 0)
            goto error;
        return ret;
    }

    virReportUnsupportedError();

 error:
    virDispatchError(domain->conn);
    return -1;
}
/*此时driver是remote，因此在remote_driver.c中找到domainResume注册
 *.domainResume = remoteDomainResume,
 */

remoteDomainResume这个是自动生成的函数，在libvirt-debuginfo包中可以看到源代码

/usr/src/debug/libvirt-3.2.0/src/remote/remote_client_bodies.h
static int
remoteDomainResume(virDomainPtr dom)
{
    int rv = -1;
    struct private_data *priv = dom->conn->privateData;
    remote_domain_resume_args args;

    remoteDriverLock(priv);

    make_nonnull_domain(&args.dom, dom);

    if (call(dom->conn, priv, 0, REMOTE_PROC_DOMAIN_RESUME,
             (xdrproc_t)xdr_remote_domain_resume_args, (char *)&args,
             (xdrproc_t)xdr_void, (char *)NULL) == -1) {
        goto done;
    }

    rv = 0;

done:
    remoteDriverUnlock(priv);
    return rv;
}

而剩下就比较简单call–>callFull–>virNetClientSendWithReply–>virNetClientSendInternal

向socket发消息，等待服务端处理，然后处理返回消息。

python client调用

python调用需要安装libvirt-python，调用代码示例：

import libvirt
conn = libvirt.open("qemu+tcp://127.0.0.1/system")
dom = conn.lookupByUUIDString("aaba00b2-e2cf-4354-b81f-82e7e3eb775e")
print dom.resume()
print dom.memoryStats()

在libvirt.py中

def resume(self):
    """Resume a suspended domain, the process is restarted from the state where
    it was frozen by calling virDomainSuspend().
    This function may require privileged access
    Moreover, resume may not be supported if domain is in some
    special state like VIR_DOMAIN_PMSUSPENDED. """
    ret = libvirtmod.virDomainResume(self._o)
    if ret == -1: raise libvirtError ('virDomainResume() failed', dom=self)
    return ret

libvirtmod是python调用c的一种实现方式，virDomainResume即与virsh中调用的函数是一致的，流程一致。

libvirt服务端处理

所有的api入口都在remote_dispatch.h中，相当于libvirt对面暴露的接口，这个文件也是libvirt编译生成的，以resume为例：

(gdb) bt
#0  qemuMonitorJSONCommandWithFd (mon=mon@entry=0x7fb090041c70, cmd=cmd@entry=0x7fb08c0145b0, scm_fd=scm_fd@entry=-1, reply=reply@entry=0x7fb09d210920) at qemu/qemu_monitor_json.c:271
#1  0x00007fb05814036e in qemuMonitorJSONCommand (reply=0x7fb09d210920, cmd=0x7fb08c0145b0, mon=0x7fb090041c70) at qemu/qemu_monitor_json.c:330
#2  qemuMonitorJSONStartCPUs (mon=0x7fb090041c70, conn=<optimized out>) at qemu/qemu_monitor_json.c:1241
#3  0x00007fb05812ba7b in qemuMonitorStartCPUs (mon=<optimized out>, conn=conn@entry=0x7fb084019660) at qemu/qemu_monitor.c:1645
#4  0x00007fb05810f826 in qemuProcessStartCPUs (driver=driver@entry=0x7fb034109750, vm=0x7fb0840008c0, conn=0x7fb084019660, reason=reason@entry=VIR_DOMAIN_RUNNING_UNPAUSED, 
    asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE) at qemu/qemu_process.c:2787
#5  0x00007fb058168fd7 in qemuDomainResume (dom=0x7fb08c009d70) at qemu/qemu_driver.c:2103
#6  0x00007fb0ad304abc in virDomainResume (domain=domain@entry=0x7fb08c009d70) at libvirt-domain.c:679
#7  0x00005593a8b184fe in remoteDispatchDomainResume (server=0x5593aaa138b0, msg=0x5593aaa3a500, args=<optimized out>, rerr=0x7fb09d210c10, client=0x5593aaa3dfe0) at remote_dispatch.h:9132
#8  remoteDispatchDomainResumeHelper (server=0x5593aaa138b0, client=0x5593aaa3dfe0, msg=0x5593aaa3a500, rerr=0x7fb09d210c10, args=<optimized out>, ret=0x7fb08c00b070)
    at remote_dispatch.h:9108
#9  0x00007fb0ad3850d2 in virNetServerProgramDispatchCall (msg=0x5593aaa3a500, client=0x5593aaa3dfe0, server=0x5593aaa138b0, prog=0x5593aaa37870) at rpc/virnetserverprogram.c:437
#10 virNetServerProgramDispatch (prog=0x5593aaa37870, server=server@entry=0x5593aaa138b0, client=0x5593aaa3dfe0, msg=0x5593aaa3a500) at rpc/virnetserverprogram.c:307
#11 0x00005593a8b2b14d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x5593aaa138b0) at rpc/virnetserver.c:148
#12 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x5593aaa138b0) at rpc/virnetserver.c:169
#13 0x00007fb0ad268b81 in virThreadPoolWorker (opaque=opaque@entry=0x5593aaa133e0) at util/virthreadpool.c:167
#14 0x00007fb0ad267f08 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#15 0x00007fb0aa449e25 in start_thread () from /lib64/libpthread.so.0
#16 0x00007fb0aa17734d in clone () from /lib64/libc.so.6

代码示例

/usr/src/debug/libvirt-3.2.0/daemon/remote_dispatch.h
static int remoteDispatchDomainResume(
    virNetServerPtr server ATTRIBUTE_UNUSED,
    virNetServerClientPtr client,
    virNetMessagePtr msg ATTRIBUTE_UNUSED,
    virNetMessageErrorPtr rerr,
    remote_domain_resume_args *args)
{
    int rv = -1;
    virDomainPtr dom = NULL;
    struct daemonClientPrivate *priv =
        virNetServerClientGetPrivateData(client);

    if (!priv->conn) {
        virReportError(VIR_ERR_INTERNAL_ERROR, "%s", _("connection not open"));
        goto cleanup;
    }

    if (!(dom = get_nonnull_domain(priv->conn, args->dom)))
        goto cleanup;

    if (virDomainResume(dom) < 0)
        goto cleanup;

    rv = 0;

cleanup:
    if (rv < 0)
        virNetMessageSaveError(rerr);
    virObjectUnref(dom);
    return rv;
}

同样是调用virDomainResume，和virsh调用不同的是：

conn->driver->domainResume
/*此时driver是qemuHypervisorDriver，因此在qemu_driver.c中找到domainResume注册
 *.domainResume = qemuDomainResume,
 */

剩下的调用过程就比较简单了，处理完业务逻辑，最终会向虚拟机socket发消息，让qemu来接收处理。

static int
qemuMonitorJSONCommandWithFd(qemuMonitorPtr mon,
                             virJSONValuePtr cmd,
                             int scm_fd,
                             virJSONValuePtr *reply)
{
    ...

    if (!(cmdstr = virJSONValueToString(cmd, false)))
        goto cleanup;
    if (virAsprintf(&msg.txBuffer, "%s\r\n", cmdstr) < 0)
        goto cleanup;
    msg.txLength = strlen(msg.txBuffer);
    msg.txFD = scm_fd;

    VIR_DEBUG("Send command '%s' for write with FD %d", cmdstr, scm_fd);
    /*通过socket发送qmp命令*/
    ret = qemuMonitorSend(mon, &msg);

    ...
    return ret;
}

qemu端处理

qemu端接收处理也比较简单，还是以resume为例，在libvirt中会调用cont命令发下给qemu，qemu则会调用qmp_cont函数，其实所有的命令都是调用qmp_func_name函数，剩余的就是具体的业务逻辑了。

(gdb) bt
#0  0x0000556179648490 in qmp_cont (errp=errp@entry=0x7ffe98da7040) at qmp.c:166
#1  0x000055617963ccea in qmp_marshal_cont (args=<optimized out>, ret=<optimized out>, errp=0x7ffe98da7088) at qmp-marshal.c:1303
#2  0x000055617981f159 in qmp_dispatch (errp=0x7ffe98da7080, request=0x55617dc87600, cmds=0x556179e1f420 <qmp_commands>) at qapi/qmp-dispatch.c:104
#3  0x000055617981f159 in qmp_dispatch (cmds=0x556179e1f420 <qmp_commands>, request=request@entry=0x55617dc87600) at qapi/qmp-dispatch.c:131
#4  0x000055617955c1e7 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.9.0/monitor.c:3850
#5  0x00005561798245e8 in json_message_process_token (lexer=0x55617b4eff88, input=0x55617b4f4040, type=JSON_RCURLY, x=36, y=24) at qobject/json-streamer.c:105
#6  0x0000556179840d1b in json_lexer_feed_char (lexer=lexer@entry=0x55617b4eff88, ch=125 '}', flush=flush@entry=false) at qobject/json-lexer.c:319
#7  0x0000556179840dde in json_lexer_feed (lexer=0x55617b4eff88, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:369
#8  0x00005561798246a9 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:124
#9  0x000055617955a77b in monitor_qmp_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.9.0/monitor.c:3893
#10 0x00005561797dbced in tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=<optimized out>) at chardev/char-socket.c:414
#11 0x00007f13aacd7969 in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#12 0x0000556179829dbc in main_loop_wait () at util/main-loop.c:213
#13 0x0000556179829dbc in main_loop_wait (timeout=<optimized out>) at util/main-loop.c:261
#14 0x0000556179829dbc in main_loop_wait (nonblocking=nonblocking@entry=0) at util/main-loop.c:517
#15 0x000055617951b18c in main () at vl.c:1909
#16 0x000055617951b18c in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4733

补充说明

All content is licensed under CC BY-NC-SA

Buit with Jekyll and 3-Jekyll theme • Hosted on Github

Table of Contents