| authors | state |
|---|---|
Domagoj Stolfa <[email protected]> |
draft |
DTrace has previously been used to trace one operating system at a time, on that operating system. With the increase of interest in distributed tracing, it would be beneficial to be able to specify which operating systems one wishes to trace in a concise manner. Currently, the DTrace compiler has no way of knowing which instance one wishes to trace.
One argument can be made that one could specify a machine to trace in the following manner:
# dtrace -n 'fbt$machine_name:::entry'
However, this is not a concise way to specify multiple providers to trace and presents a problem with userland probes.
The problem can be observed on the given example:
# dtrace -n 'pid$target:libc:strcpy:entry'
There is no clear way of representing which operating systems would be traced in this case.
A new tuple entry will be defined in the probe name. The user may or may not use
this tuple entry in order to specify which probes to attach to. Furthermore, a
command line parameter is added to the dtrace command to specify an instance.
The interface is detailed in the following sections
###instance tuple entry
In the current state of DTrace, the user can specify a 4-tuple in the form of
provider:module:function:name. The new tuple entry will operate in a similar
manner to the existing tuple entries. The user need not specify this tuple
entry, in which case DTrace defaults to the host instance, tracing the current
machine in a backwards-compatible way.
###-M command line argument
One might want to attach to every probe on a given instance. The -M command
line flag, which stands for "machine" allows to do so.
###-l flag changes
When listing the probes, it is beneficial to know which probe is located on which instance.
# dtrace -n '*:syscall:::entry { @[execname] = count(); }'
dtrace: description '*:syscall:::entry ' matched 1084 probes
# dtrace -n 'syscall:::entry { @[execname] = count(); }'
dtrace: description 'syscall:::entry ' matched 1084 probes
# dtrace -n 'host:syscall:::entry { @[execname] = count(); }'
dtrace: description 'host:syscall:::entry ' matched 1084 probes
# dtrace -M host
dtrace: description 'host' matched 66356 probes
# dtrace -M 'foo'
dtrace: invalid probe specifier foo: probe description foo:::: does not match any
probes
# dtrace -n 'nomatch:syscall:::entry'
dtrace: invalid probe specifier nomatch:syscall:::entry probe description
nomatch:syscall:::entry does not match any probes
The output of # dtrace -l is visible in this document.
It is necessary to add this tuple entry to the DOF. Changes have to be made in
dt_dof.c and dt_subr.c.
Each provider should be associated with an instance. This is required in order to stay backwards-compatible with each of the existing providers.
A new linked list structure is added, dtrace_instance_t. It is defined as
follows:
typedef struct dtrace_instance {
char *dtis_name; /* instance name */
struct dtrace_provider *dtis_provhead; /* first provider in the instance */
struct dtrace_instance *dtis_next; /* next instance */
struct dtrace_instance *dtis_prev; /* previous instance */
} dtrace_instance_t;
Each instance has a list of it's providers. This is required in order to have separation of providers in different instances, especially during deletion.
Furthermore, an additional hash table is added, dtrace_byinstance and operates
similary to dtrace_bymod, dtrace_byfunc, and dtrace_byname. This is done
to identify which probes will be installed.
Additionally, a new DIF variable is needed, DIF_VAR_PROBEISTC, which operates
similarly to DIF_VAR_PROBEPROV, DIF_VAR_PROBEMOD, DIF_VAR_PROBEFUNC and
DIF_VAR_PROBENAME. This is necessary for DIF variable lookups.
Additionally, dtrace_probe_t now contains the instance name and pointer to the
next and previous probe in the instance hash.
struct dtrace_probe {
dtrace_id_t dtpr_id; /* probe identifier */
dtrace_ecb_t *dtpr_ecb; /* ECB list; see below */
dtrace_ecb_t *dtpr_ecb_last; /* last ECB in list */
void *dtpr_arg; /* provider argument */
dtrace_cacheid_t dtpr_predcache; /* predicate cache ID */
int dtpr_aframes; /* artificial frames */
dtrace_provider_t *dtpr_provider; /* pointer to provider */
char *dtpr_instance; /* probe's instance name */
char *dtpr_mod; /* probe's module name */
char *dtpr_func; /* probe's function name */
char *dtpr_name; /* probe's name */
dtrace_probe_t *dtpr_nextinstance; /* next in instance hash */
dtrace_probe_t *dtpr_previnstance; /* previous in instance hash */
dtrace_probe_t *dtpr_nextmod; /* next in module hash */
dtrace_probe_t *dtpr_prevmod; /* previous in module hash */
dtrace_probe_t *dtpr_nextfunc; /* next in function hash */
dtrace_probe_t *dtpr_prevfunc; /* previous in function hash */
dtrace_probe_t *dtpr_nextname; /* next in name hash */
dtrace_probe_t *dtpr_prevname; /* previous in name hash */
dtrace_genid_t dtpr_gen; /* probe generation ID */
};
Similarly to dtrace_probe_t, dtrace_probekey_t now contains the instance
name and the instance matching function.
typedef struct dtrace_probekey {
char *dtpk_instance; /* instance name to match */
dtrace_probekey_f *dtpk_imatch; /* instance matching function */
char *dtpk_prov; /* provider name to match */
dtrace_probekey_f *dtpk_pmatch; /* provider matching function */
char *dtpk_mod; /* module name to match */
dtrace_probekey_f *dtpk_mmatch; /* module matching function */
char *dtpk_func; /* func name to match */
dtrace_probekey_f *dtpk_fmatch; /* func matching function */
char *dtpk_name; /* name to match */
dtrace_probekey_f *dtpk_nmatch; /* name matching function */
dtrace_id_t dtpk_id; /* identifier to match */
} dtrace_probekey_t;
Furthermore, dtrace_register() is required to be implemented with a new, more
generic function, dtrace_distributed_register(). This is because one needs to
pass in the instance name when registering a provider. The function
dtrace_register() would simply pass in "host" as the instance to allow for
backwards compatibility.
Because of that, a new mutex needs to be introduced, dtrace_instance_lock. It
is held at the same times as dtrace_provider_lock in
dtrace_distributed_register(), dtrace_module_loaded(), dtrace_open(),
dtrace_ioctl() and dtrace_load(). Additionally, it needs to be held in dtrace_unregister()
when an instance is being looked up during provider deletion.
The impact on existing systems is seen in the possibility to specify a new tuple entry, a new command line argument -M and the way help is displayed for DTrace with the addition of new command line argument.
Various error messages will now print out the instance name if the user has specified one, and otherwise omit "host", so that the message will be identical to the existing ones.
Running # dtrace -l now outputs the instance name as well, in the standard way
that DTrace has been outputting the existing 4 tuple entries.
The addition alone presents no security vulnerabilities to the OpenDTrace system, however in the context it would be used, it is important to satisfy the right security propreties and establish an information flow correctly.