Skip to content

Latest commit

 

History

History
1231 lines (1006 loc) · 66.1 KB

README.md

File metadata and controls

1231 lines (1006 loc) · 66.1 KB

How to: demo

Intro

We're going to use a single Demo application, which we'll evolve and extend in order to briefly show the capabilities of Metac.

Makefile system

Metac uses DWARF information to produce a reflection base. While other platforms work similarly, macOs has a different tooling implementation - DWARF information can be generated only for the executable binary. It's also necessary to explicitly call dsymutil to produce DWARF. Nevertheless other supported platforms don't have this, for consistency we're building the application first time with flags -g3 -D_METAC_OFF_, collect DWARF data, generate an additional C file with reflect information and rebuild the final application that includes everything. The build process may look a bit too complex, but we automate that in Makefile to simplify it. This is the reason why we will start the demo from writing it's Makefile. The chapter about Makefiles will briefly tell what capabilities are included. Also the Makefile system supports another feature - Golang-like unit-tests.

Unit-tests

Golang has a pretty strong support of writing unit-tests. Metac needed unit-tests to check implementation of the API. The similar approach with *_test.c was used to build unit-tests. The Makefile subsystem supports building and running when. The dedicated chapter will describe how to use it. The next chapters will be built not only on building the Demo application itself, but also demonstrating examples on Unit-tests - in some cases it is more efficient.

Reflection

Reflection implementation for C has some C-specific complications. The chapter will be dedicated to a brief description of those aspects. It will provide some examples of how to use Metac Reflection API.

Platforms

Metac in general requires Golang installed on the system. This is to build the binary which reads DWARF and generates C files with metainformation. The libmetac doesn't require any dependencies. Check package is needed to be able to run unit-tests. Pkg-config package is needed by Makefile in order to identify which CFLAGS/LDFLAGS are needed for the check library.

Integrational testing is running for Linux, macOS and Windows: check the project settings.

MacOS

Install developers tools:

xcode-select --install

Install Golang.

Install brew and using brew install:

brew install pkg-config check

Ubuntu

Install Golang and the following packages:

apt-get install pkg-config check

Optionally (recommended for development, but Metac works without them) install:

apt-get install valgrind gcovr

Windows (msys2)

Install Golang.

Install msys2.

Install packages:

packman -S base-devel git pkg-config mingw-w64-x86_64-check mingw-w64-x86_64-toolchain

Demo

Let's assume that we already have some C-code which is written without reflection and now we want to wrap it with some Metac features. Folder step_00 contains the initial version of the demodb library header, implementation and main.c which uses that library and which we'll use throughout this document.

It even has some simple Makefile which can build and clean this demo application.

Rewriting Makefile

It's not mandatory, but in this demo we're going to use Metac Makefile support of different features, and because of that we'll need to translate Makefile to Metac-compliant.

The original file was really simple:

all: demodb

demodb: main.o demodb.o

clean:
	rm demodb *.o

Here we're building a binary application file from to object files.

Metac uses a very similar to KBUILD approach, but with some additional specifics. As well as KBUILD it calls the external makefile and points to its own location so the main Makefile could use some data from it. Here is the corresponding part:

ifeq ($(M),)
#METAC_ROOT=path to metac

all: test target

target:
	$(MAKE) -C $(METAC_ROOT) M=$(PWD) target

clean:
	$(MAKE) -C $(METAC_ROOT) M=$(PWD) clean

test:
	$(MAKE) -C $(METAC_ROOT) M=$(PWD) test

.PHONY: all clean test
endif

This part looks exactly like a Kbuild module. Metac even took the same Module-path parameter name $(M).

The next part of the file is a bit different from Linux kernel Kbuild format, but still very close:

rules+= \
	target \
	demodb

TPL-target:=phony_target
IN-target:= \
	demodb

TPL-demodb:=bin_target
IN-demodb= \
	main.o \
	demodb.o

Every Metac module Makefile must have a variable called rules, which lists all Makefile rules which will be generated by this Makefile. In this particular case we have 2 rules: target and demodb. Please pay attention that the first part already referenced this part:

all: ... target

target:
	$(MAKE) -C $(METAC_ROOT) M=$(PWD) target

This is exaclty the same Makefile goal name.

TPL-target:=phony_target
IN-target:= \
	demodb

The next 2 lines say that target is actually a .PHONY type of target and it depends on demodb. It's possible to list here many dependencies in addition to demodb. Next part is:

TPL-demodb:=bin_target
IN-demodb= \
	main.o \
	demodb.o

This construction tells that demodb is an application binary rule and that we have to build it from 2 object files. Similar to what we saw in the original Makefile.

In order to run the build process we can do from the step_01 folder:

step_01 % make METAC_ROOT=../../..
make -C ../../.. M=/home/test/metac/doc/demo/step_01 test
All test dependencies were: bin_test module_test
make -C ../../.. M=/home/test/metac/doc/demo/step_01 target
cc -I./include -c -MMD -MF /home/test/metac/doc/demo/step_01/main.d -MP -MT '/home/test/metac/doc/demo/step_01/main.o /home/test/metac/doc/demo/step_01/main.d' -o /home/test/metac/doc/demo/step_01/main.o /home/test/metac/doc/demo/step_01/main.c
cc -I./include -c -MMD -MF /home/test/metac/doc/demo/step_01/demodb.d -MP -MT '/home/test/metac/doc/demo/step_01/demodb.o /home/test/metac/doc/demo/step_01/demodb.d' -o /home/test/metac/doc/demo/step_01/demodb.o /home/test/metac/doc/demo/step_01/demodb.c
cc /home/test/metac/doc/demo/step_01/main.o /home/test/metac/doc/demo/step_01/demodb.o  -o /home/test/metac/doc/demo/step_01/demodb
Built dependencies:

The advantage of this build approach is that if we now update any file which is dependency (e.g. header file) make will rebuild the related files including the final target. Just do touch demodb.h && make METAC_ROOT=../../... The automatically generated dependencies are stored in *.d files.

Another difference from the original Makefile is that there is no clean rule. It's getting generated automatically (the similar approach is used in Kbuild).

% make METAC_ROOT=../../.. clean
make -C ../../.. M=/home/test/metac/doc/demo/step_01 clean
echo /home/test/metac/doc/demo/step_01/demodb /home/test/metac/doc/demo/step_01/main.o /home/test/metac/doc/demo/step_01/demodb.o /home/test/metac/doc/demo/step_01/main.d /home/test/metac/doc/demo/step_01/demodb.d | xargs -n 1| sort -u| xargs rm
% ls *
Makefile	demodb.c	demodb.h	main.c

It's possible to refer to bin.mk for information about additional parameters the bin rule template accepts:

  1. IN-<rulename> - list of object files to be linked into the binary
  2. LDFLAGS-<rulename> - to set the rule specific linker flags
  3. DEPS-<rulename> - dependency rule which has to be built, but which won't be listed to linker automatically. This can be used e.g. to build some .so or .a target first and link it to this binary. in that case it's necessary to use this parameter in combination with LDFLAGS-<rulename>
  4. PRE-<rulename>/POST-<rulename> - used to wrap linker with some pre or post commands

As a result we should get Makefile which is located in folder step_01.

Adding empty unit-tests and running it

Many experienced developers use the following approach: the code which isn't tested must be considered as 'non-working'. It's always a good idea to cover some logical modules by unit-tests. In our case we have demodb code which we will try to cover in this chapter.

The approach used in Metac is similar to golang - just create a file *_test.c and it will be considered by Makefile as a file with test. Here is the smallest content:

#include "metac/test.h"

METAC_START_TEST(sometest) {
    /* put here the test code */
} END_TEST

This code uses check library under the hood. It's necessary to have it installed on the host along with pkg-config to make this work. pkg-config is used by Makefile to identify what CFLAGS/LDFLAGS is necessary to use for this particular host platform.

No changes are needed for Makefile. To run this test we'll need just to run make test METAC_ROOT=<path to the metac Makefile>:

step_02 % make METAC_ROOT=../../.. test
make -C ../../.. M=/home/test/metac/doc/demo/step_02 test
cc -I./include -g3 -Wno-format-extra-args --coverage -D_THREAD_SAFE -I/opt/homebrew/Cellar/check/0.15.2/include -c -MMD -MF /home/test/metac/doc/demo/step_02/demodb_test.d -MP -MT '/home/test/metac/doc/demo/step_02/demodb_test.o /home/test/metac/doc/demo/step_02/demodb_test.d' -o /home/test/metac/doc/demo/step_02/demodb_test.o /home/test/metac/doc/demo/step_02/demodb_test.c
cc -I./include -g3 -D_THREAD_SAFE -I/opt/homebrew/Cellar/check/0.15.2/include -Wno-format-extra-args -D_THREAD_SAFE -I/opt/homebrew/Cellar/check/0.15.2/include -g3  -D_METAC_OFF_ -c -MMD -MF /home/test/metac/doc/demo/step_02/demodb_test.meta.d -MP -MT '/home/test/metac/doc/demo/step_02/demodb_test.meta.o /home/test/metac/doc/demo/step_02/demodb_test.meta.d' -o /home/test/metac/doc/demo/step_02/demodb_test.meta.o /home/test/metac/doc/demo/step_02/demodb_test.c
cc -I./include -g3 -D_THREAD_SAFE -I/opt/homebrew/Cellar/check/0.15.2/include -c -MMD -MF /home/test/metac/doc/demo/step_02/demodb_test.dummy.d -MP -MT '/home/test/metac/doc/demo/step_02/demodb_test.dummy.o /home/test/metac/doc/demo/step_02/demodb_test.dummy.d' -o /home/test/metac/doc/demo/step_02/demodb_test.dummy.o /home/test/metac/doc/demo/step_02/demodb_test.dummy.c
cc /home/test/metac/doc/demo/step_02/demodb_test.meta.o /home/test/metac/doc/demo/step_02/demodb_test.dummy.o --coverage -L/opt/homebrew/Cellar/check/0.15.2/lib -lcheck --coverage -L/opt/homebrew/Cellar/check/0.15.2/lib -lcheck -o /home/test/metac/doc/demo/step_02/./_meta_demodb_test
(which dsymutil) && dsymutil /home/test/metac/doc/demo/step_02/./_meta_demodb_test || echo "Couldn't find dsymutil"
/usr/bin/dsymutil
go build
./metac run metac-test-gen -s 'path_type: "macho"' -s 'path: "/home/test/metac/doc/demo/step_02/./_meta_demodb_test"' > /home/test/metac/doc/demo/step_02/demodb_test.test.c
cc -I./include -g3 -D_THREAD_SAFE -I/opt/homebrew/Cellar/check/0.15.2/include -c -MMD -MF /home/test/metac/doc/demo/step_02/demodb_test.test.d -MP -MT '/home/test/metac/doc/demo/step_02/demodb_test.test.o /home/test/metac/doc/demo/step_02/demodb_test.test.d' -o /home/test/metac/doc/demo/step_02/demodb_test.test.o /home/test/metac/doc/demo/step_02/demodb_test.test.c
./metac run metac-reflect-gen -s 'path_type: "macho"' -s 'path: "/home/test/metac/doc/demo/step_02/./_meta_demodb_test"' > /home/test/metac/doc/demo/step_02/demodb_test.reflect.c
cc -I./include -c -MMD -MF /home/test/metac/doc/demo/step_02/demodb_test.meta.d -MP -MT '/home/test/metac/doc/demo/step_02/demodb_test.reflect.o /home/test/metac/doc/demo/step_02/demodb_test.meta.d' -o /home/test/metac/doc/demo/step_02/demodb_test.reflect.o /home/test/metac/doc/demo/step_02/demodb_test.reflect.c
cc /home/test/metac/doc/demo/step_02/demodb_test.o /home/test/metac/doc/demo/step_02/demodb_test.test.o /home/test/metac/doc/demo/step_02/demodb_test.reflect.o --coverage -L/opt/homebrew/Cellar/check/0.15.2/lib -lcheck -o /home/test/metac/doc/demo/step_02/demodb_test
Running suite(s): /home/test/metac/doc/demo/step_02/demodb_test
100%: Checks: 1, Failures: 0, Errors: 0
All test dependencies were: bin_test module_test
step_02 %

There are lots of things happened here:

  1. build of demodb_test.o from demodb_test.c
  2. build of demodb_test.meta.o from demodb_test.c. Note: *.meta.o files are the object-files that are ALWAYS built with options -g3 -D_METAC_OFF_. This is part of the build process for the binary from which DWARF info will be taken.
  3. generation of demodb_test.dummy.c which contains an empty main function. This is needed, because without that we can't build any binary.
  4. compilation of demodb_test.dummy.o
  5. linking of _meta_demodb_test application binary. This binary won't work because it has an empty main function. but it can be used to collect DWARF information.
  6. since we ran on macOs we needed to run dsymutil _meta_demodb_test in order to access DWARF information
  7. metac golang binary was built (because it was the first run and it didn't present). it's possible to set METAC path if you want to use the external binary.
  8. we used run metac-test-gen command to generate demodb_test.test.c which will contain the actual main function for the test. This file is used instead of demodb_test.dummy on the second pass of build.
  9. we used run metac-reflect-gen to generate a reflection db for the test. Actually it wasn't necessary for this particular case because we didn't use reflection in the test. To avoid that step we had to add something like REFLECT-<testname>=n to Makefile. for our case REFLECT-demodb_alt_test:=n Though reflection information won't hurt for testing.
  10. building and running demodb_test. It's seen that the test has passed ok - it was empty.

for the reference, here is generated demodb_test.test.c file:

#include <check.h>  /* all */
#include "metac/reqresp.h"

/* early declarations of requests */
extern const TTest * METAC_REQUEST(test, sometest);

int main(int argc, char **argv) {
    Suite *s = suite_create(argc>0?argv[0]:(__FILE__));
    TCase *tc = tcase_create("default");

    /* run tests  */
    tcase_add_test(tc, METAC_REQUEST(test, sometest));

    suite_add_tcase(s, tc);
    SRunner *sr = srunner_create(s);
    srunner_set_fork_status(sr, CK_FORK_GETENV);
    srunner_run_all(sr, CK_ENV);

    int f = srunner_ntests_failed(sr);
    srunner_free(sr);
    return (f == 0) ? 0 : 1;
}

As you can see - it uses check library under the hood, but it's not necessary to write this entrypoint manually. To generate this file metac-test-gen go-template module was used.

Unit-tests are always built with coverage information. For our simple case we can run make METAC_ROOT=../../.. test RUNMODE=coverage, and we'll see the coverage for this particular file. But for more complex cases where we have many tests for many C-files and we need to get complete coverage information for each of the files - please install gcovr and use gcovr -p ./ -e '.*_test\.c|.*_checkmk\.c|.*\.h' or similar. It may even build html files and show coverage for each line.

Since C requires the developer to manage dynamic memory it's very useful to check test for memory leaks. On Linux it's possible to install valgrind package and after that run the test with the following command: make METAC_ROOT=../../.. test RUNMODE=valgrind. This will show if the test has any leakage.

One more important note is - if there are many test files created it's possible to run only selected ones. make test supports optional parameters INCLIDE XOR EXCLUDE which may be used to set a pattern in Makefile format (e.g. you need to use % instead of *) of the tests which are necessary to run. The implementation of tests related rules can be found in test.mk.

Writing dbdemo unit-tests

Now when we created the dbdemo_test.c file with an empty single test, lets try to write some actual unit-test code.

The idea is really the same as for Golang unit-tests, e.g.:

#include "metac/test.h"

#include "demodb.c"

METAC_START_TEST(append_test) {
    struct {
        person_t * p_in;
        int expected_err;
    }tcs[] = {
        {
            .p_in = NULL,
            .expected_err = 1,
        },
        {
            .p_in = (person_t[]){{
                .firstname="Joe",
                .lastname="Doe",
                .age = 43,
                .marital_status = msMarried,
            }},
            .expected_err = 0,
        },
        {
            .p_in = (person_t[]){{
                .firstname="Jane",
                .lastname="Doe",
                .age = 34,
                .marital_status = msMarried,
            }},
            .expected_err = 0,
        },
        {
            .p_in = (person_t[]){{
                .firstname="Jack",
                .lastname="Doe",
                .age = 3,
                .marital_status = msSingle,
            }},
            .expected_err = 0,
        },
    };
    db_t * p_db =  new_db();
    for (int tc_inx = 0; tc_inx < sizeof(tcs)/sizeof(tcs[0]); tc_inx++) {
        int res = db_append(&p_db, tcs[tc_inx].p_in);
        fail_unless((res != 0) == (tcs[tc_inx].expected_err != 0), "unexpected err result %i, expected %i", 
            res, tcs[tc_inx].expected_err);
    }
    db_delete(p_db);
}END_TEST

First - pay attention that the test does #include "demodb.c". That allows you to get access to the whole module and test static functions if needed as well. As an alternative it's possible to only include header file. But in that case it's necessary to link the test with the object file and Makefile will need modifications. There is demodb_alt_test.c where we have the only difference - usage of header file instead. The corresponding changes in Makefile to support this approach are:

rules+= libdemodb.a
TPL-libdemodb.a:=a_target
IN-libdemodb.a=demodb.o

#tests
DEPS-demodb_alt_test:=$(M)/libdemodb.a
LDFLAGS-demodb_alt_test:=-L$(M) -ldemodb
REFLECT-demodb_alt_test:=n

Here we're building a library .a file and linking it to this test. The last line is - not to generate reflection meta-information.

Check library has an utility called checkmk, which accepts a special file format as input and generates test that inludes tests. We also support that: such files must have name pattern *.checkmk. Metac will generate test binary with the name *_checkmk for such file. Here is the converted example. And we need to make the corresponding changes in Makefile:

DEPS-demodb_alt_checkmk:=$(M)/libdemodb.a
LDFLAGS-demodb_alt_checkmk:=-L$(M) -ldemodb
REFLECT-demodb_alt_checkmk:=n

To make sure that this works ok - just run make METAC_ROOT=../../.. test or make METAC_ROOT=../../.. in the folder step_02

Reflection

We already mentioned several times that Metac is very similar in some aspects to Golang. Reflection isn't exception. There are ways to get information about Type of global or local variable. It's also possible to get information about function definition: returning type and arguments.

There are 2 more levels of API - work with individual memory object values: identifying type, getting/setting values based on the specific type, etc. And the last level - is deep functions. On this level we may see a C-specific issue - ambiguity on what to expect as data in some special cases. Though there are mechanisms to overcome that and make deep function work even for those cases.

Let's start from the first level:

Requesting information about type

The type information is returned by the structure called metac_entry_t. The set of API functions related to that structure can be found in include/metac/reflect/entry.h.

Let's quickly write another unit-test which will demonstrate how to get type information.

There are several approaches that can be used to get information about types:

  1. via so-called 'links'
  2. via so-called 'declocs' (or declaration locations);
  3. via db which allows you to get info via name. - We're not going to cover this, since it's not a very convenient way at least for now.
Requesting information about type via links

This approach works only for global variables and functions. It can be used if you want to take information from some external code - some code that you can compile, but it's written as a module and you don't want to modify it.

In order to get type information - just have the header file included. Here is the first step which is needed to get information about some type (see main.c in step_03):

#include "metac/reflect.h"

person_t * p_person = NULL;
METAC_GSYM_LINK(p_person);

We're including a reflection header, defining some variable and using its name in METAC_GSYM_LINK. It's ok to separate usage of macro and variable definition. e.g. p_person potentially could be located in the module itself. We can do the same for some functions from our library.

METAC_GSYM_LINK(db_append);

In order to get now the type information we can do the following in some of the functions:

	metac_entry_t *p_person_entry = METAC_GSYM_LINK_ENTRY(p_person);
	if (p_person_entry == NULL) {
		printf("wasn't able to get p_person_entry type information\n");
		return 1;
	}
	metac_entry_t *p_db_append_entry = METAC_GSYM_LINK_ENTRY(db_append);
	if (p_person_entry == NULL) {
		printf("wasn't able to get db_append type information\n");
		return 1;
	}

if we can now try to compile we'll get an error:

step_03 % make METAC_ROOT=../../..
make -C ../../.. M=/home/test/metac/doc/demo/step_03 test
Running suite(s): /home/test/metac/doc/demo/step_03/demodb_test
100%: Checks: 3, Failures: 0, Errors: 0
All test dependencies were: bin_test module_test
make -C ../../.. M=/home/test/metac/doc/demo/step_03 target
cc -I./include -c -MMD -MF /home/test/metac/doc/demo/step_03/main.d -MP -MT '/home/test/metac/doc/demo/step_03/main.o /home/test/metac/doc/demo/step_03/main.d' -o /home/test/metac/doc/demo/step_03/main.o /home/test/metac/doc/demo/step_03/main.c
cc /home/test/metac/doc/demo/step_03/main.o /home/test/metac/doc/demo/step_03/demodb.o  -o /home/test/metac/doc/demo/step_03/demodb
Undefined symbols for architecture arm64:
  "_metac_dflt_gsym_db_append", referenced from:
      _metac__dflt_gsym_db_append in main.o
  "_metac_dflt_gsym_p_person", referenced from:
      _metac__dflt_gsym_p_person in main.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [/home/test/metac/doc/demo/step_03/demodb] Error 1

This is expected - our makefile isn't created to generate reflection information yet. Let's modify it:

TPL-demodb:=bin_target
IN-demodb= \
	main.o \
	demodb.o \
	demodb.reflect.o
LDFLAGS-demodb=-Lsrc -lmetac
DEPS-demodb=src/libmetac.a

#added to support demodb.reflect.o
rules+= \
	_meta_demodb \
	demodb.reflect.c

TPL-_meta_demodb=bin_target
IN-_meta_demodb= \
	main.meta.o \
	demodb.meta.o
LDFLAGS-_meta_demodb=-Lsrc -lmetac
POST-_meta_demodb=$(METAC_POST_META)

TPL-demodb.reflect.c:=metac_target
IN-demodb.reflect.c=_meta_demodb
METACFLAGS-demodb.reflect.c=run metac-reflect-gen $(METAC_OVERRIDE_IN_TYPE)

First - we added demodb.reflect.o to the demodb binary object files and lining src/libmetac.a which contains all metac library functions. The rest of the added lines are needed to generate demodb.reflect.c. For doing this we're defining 2 more rules.

First one _meta_demodb is our intermittent binary which is built of main.c and demodb.c object files, which are compiled with proper flags to get debug information. Also we're making sure to call dsymutil right after we built this binary by the line POST-_meta_demodb=$(METAC_POST_META).

Second rule is - how we're building demodb.reflect.c. We're using the metac tool, running it on _meta_demodb and providing arguments run metac-reflect-gen $(METAC_OVERRIDE_IN_TYPE). Here metac-reflect-gen is a name of module which may be found in module/metac-reflect-gen folder. Metac allows setting several folders where modules can be found with -M arguments which behave similar to -I in gcc/clang. That allows to extend functionality and create application specific modules which can generate arbitrary test files using go-template. $(METAC_OVERRIDE_IN_TYPE) is needed here to instruct the module about the format of the input binary. That could be elf, macho, pe, yaml or json - the set that is supported by the metac-reflect-gen module.

Now if we build the application - it will execute and exit without error. What can we do about the type information? For the sake of the demo, let's just print C representation of those objects. We can use metac_entry_cdecl function. Don't forget to free its result, because it generates the string in dynamic memory.

We're adding #include <stdlib.h> in the beginning and the following code after the original one:

	char * p_cdecl = NULL;
	p_cdecl = metac_entry_cdecl(p_person_entry);
	if (p_cdecl == NULL) {
		return 1;
	}
	printf("p_person_entry: %s\n", p_cdecl);
	free(p_cdecl);
	p_cdecl = metac_entry_cdecl(p_db_append_entry);
	if (p_cdecl == NULL) {
		return 1;
	}
	printf("p_db_append_entry: %s\n", p_cdecl);
	free(p_cdecl);

Now if we compile the code and run it we'll see:

step_03 % ./demodb
p_person_entry: person_t * p_person
p_db_append_entry: int db_append(db_t * * pp_db, person_t * p_person)

As we can see the string contains not only type information, but also variable or function name. It's a good time to explain what metac_entry_t can contain.

Understanding metac_entry_t

In few words metac_entry_t is a representation of DWARF Debugging Information Entry (or DIE). Metac doesn't need all the data provided by DWARF, some of the attributes are omitted, but the idea is the same. Each of the entries has a kind, which can be taken by metac_entry_kind. All kinds are listed in metac/const.h. Depending on the kind it's possible to call one or another subgroup of API functions. e.g. for pointers (which also can be checked by metac_entry_is_pointer function) it's ok to call any of:

metac_flag_t metac_entry_is_void_pointer(metac_entry_t *p_entry);
metac_flag_t metac_entry_is_declaration_pointer(metac_entry_t *p_entry);
metac_entry_t * metac_entry_pointer_entry(metac_entry_t *p_entry);

Very important note: entry can have a kind METAC_KND_variable, but metac_entry_is_pointer will return non-zero. This is because Metac separates so called final kinds, which for C language are: METAC_KND_base_type (char, int, complex, double...), METAC_KND_pointer_type, METAC_KND_enumeration_type, METAC_KND_subroutine_type (functions), METAC_KND_array_type, METAC_KND_struct_type and METAC_KND_union_type. If it's necessary to get information about the type, use metac_entry_final_entry which will return one of those kinds. To understand more about how reflection information is represented please run dwarfdump _meta_demodb (or dwarfdump _meta_demodb.dSYM for macOs), and less demodb.reflect.c.

Let's extend our example with that example about structure of person_t type:

	if (metac_entry_is_pointer(p_person_entry)==0) {
		return 1;
	}
	metac_entry_t *p_person_typedef = metac_entry_pointer_entry(p_person_entry);
	p_cdecl = metac_entry_cdecl(p_person_typedef);
	if (p_cdecl == NULL) {
		return 1;
	}
	printf("p_person_typedef: %s\n", p_cdecl);
	free(p_cdecl);

We can see now

p_person_typedef: person_t %s

In case it's not variable or structure field or argument metac_entry_cdecl will always return a string with placeholder which is possible to change to anything with printf.

Let's go next level:

	metac_entry_t *p_person_struct = metac_entry_final_entry(p_person_typedef, NULL);
	p_cdecl = metac_entry_cdecl(p_person_struct);
	if (p_cdecl == NULL) {
		return 1;
	}
	printf("p_person_struct: ");
	printf(p_cdecl, "");
	printf("\n");
	free(p_cdecl);

as a result we're getting the string which correlates with definition:

p_person_struct: struct {char * firstname; char * lastname; int age; enum {msSingle = 0, msMarried = 1, msDivorsed = 2,} marital_status; }

Now lets switch to another method of getting data

Requesting information about type via declloc

Declloc (Declaration location) is a more advanced method which can be used either for global symbols or local variables within functions. It's called like this because this mechanism uses a location variable with special naming. When metac-reflect-gen finds such a variable it adds information about all other variables located in the same lexical block (or globally if it's global object) which are declared on the same line. This filtering is made just not to overload DB with not-requested information.

Even though declloc approach can be used for any variable, it doesn't allow to get information about function which is already declared somewhere else. That means that it doesn't overlap with links for 100% cases.

WITH_METAC_DECLLOC is a macro which makes its second parameter to be considered as 1 line of code. It's possible to use it even with very big constructions, like functions.

Lets try to show several examples in the next folder step_04:

WITH_METAC_DECLLOC(gl_dl1,
int some_function(int x) {
	return -1;
})

WITH_METAC_DECLLOC(gl_dl2, person_t * p_gl_person = NULL);

int main(){
	WITH_METAC_DECLLOC(dl, person_t * p_person = NULL;
	// We even can use METAC_ENTRY_FROM_DECLLOC inside WITH_METAC_DECLLOC
	metac_entry_t *p_person_entry = METAC_ENTRY_FROM_DECLLOC(dl, p_person));

	metac_entry_t *p_gl_some_function = METAC_ENTRY_FROM_DECLLOC(gl_dl1, some_function);
	metac_entry_t *p_gl_person_entry = METAC_ENTRY_FROM_DECLLOC(gl_dl2, p_gl_person);

	// we're getting the same type information from p_person_entry and p_gl_some_function
	if (metac_entry_final_entry(p_person_entry, NULL) == NULL ||
		metac_entry_final_entry(p_person_entry, NULL) != metac_entry_final_entry(p_gl_person_entry, NULL)) {
			return 1;
	}

	char * p_cdecl = NULL;
	p_cdecl = metac_entry_cdecl(p_person_entry);
	if (p_cdecl == NULL) {
		return 1;
	}
	printf("p_person_entry: %s\n", p_cdecl);
	free(p_cdecl);
	p_cdecl = metac_entry_cdecl(p_gl_some_function);
	if (p_cdecl == NULL) {
		return 1;
	}
	printf("p_gl_some_function: %s\n", p_cdecl);
	free(p_cdecl);

	if (metac_entry_is_pointer(p_person_entry)==0) {
		return 1;
	}
	metac_entry_t *p_person_typedef = metac_entry_pointer_entry(p_person_entry);
	p_cdecl = metac_entry_cdecl(p_person_typedef);
	if (p_cdecl == NULL) {
		return 1;
	}
	printf("p_person_typedef: %s\n", p_cdecl);
	free(p_cdecl);

	metac_entry_t *p_person_struct = metac_entry_final_entry(p_person_typedef, NULL);
	p_cdecl = metac_entry_cdecl(p_person_struct);
	if (p_cdecl == NULL) {
		return 1;
	}
	printf("p_person_struct: ");
	printf(p_cdecl, "");
	printf("\n");
	free(p_cdecl);

The output is:

step_04 % ./demodb
p_person_entry: person_t * p_person
p_gl_some_function: int some_function(int x)
p_person_typedef: person_t %s
p_person_struct: struct {char * firstname; char * lastname; int age; enum {msSingle = 0, msMarried = 1, msDivorsed = 2,} marital_status; }

There is a simple way to even get the type of any expression, not necessarily variable. Just use typeof(<expression>) * p_some_var_name = NULL; and get the type of the pointer as we did before.

Declloc approach uses a single db sorted by source file, declloc name and variable name. To see how it looks like you can do less demodb.reflect.c and find struct metac_entry_db *METAC_ENTRY_DB_NAME(dflt) = (struct metac_entry_db[]).

Work with values using reflection information

As we mentioned before - there are just few 'final' types in C:

  1. base types (char, short, int ...)
  2. enums
  3. pointers
  4. arrays
  5. structures/unions

Metac provides a set of functions to read/write/copy/compare. All of the functions are located in metac/reflect/value.h. All of the functions accept metac_value_t pointer. The object is always in dynamic memory and must be created/deleted with:

metac_value_t * metac_new_value(metac_entry_t *p_entry, void * addr);
void metac_value_delete(metac_value_t * p_val);

Value contains just 2 items: information about the type and address of the actual data where of the given type.

There are 2 macroses that wrap metac_entry_t macroses and use them to create metac_value_t.

#define METAC_VALUE_FROM_LINK(_name_) ...
#define METAC_VALUE_FROM_DECLLOC(_name_, _val_) ...

In addition to the simple set of functions to read/write/copy/compare, each of the 'final' kinds provide their own specific set of functions. Good example is - set of functions which provide access to structure field values:

/* kind = METAC_KND_structure_type || kind == METAC_KND_union_type */
metac_flag_t metac_value_has_members(metac_value_t *p_val);
metac_num_t metac_value_member_count(metac_value_t *p_val);
/* to get member name use metac_value_name(p_val)*/
metac_value_t * metac_new_value_by_member_id(metac_value_t *p_val, metac_num_t member_id);

The produced value of the member element will contain the member type and address. Similarly work:

metac_value_t * metac_new_element_count_value(metac_value_t *p_val, metac_num_t count);  /* creates new value with overridden element count. useful for flexible arrays if we got actual len*/
metac_value_t * metac_new_value_by_element_id(metac_value_t *p_val, metac_num_t element_id);

This allows it to iterate in depth of structure/array and read and write every leaf element of complex structures. We didn't demonstrate how to use those functions because they are very simple and work exactly as golang/reflect. This is exactly what deep functions do. User an use those functions directly or use the 'deep' set of functions which will be described next.

Work with 'deep' functions

Golang/reflect defines a similar set of functions. But there is one important difference between Go and C. Go is called by some people as C of the 21st century. One of the reasons we could agree is - Go architecture was able to resolve several ambiguities which C has. Let's walk through them:

  1. Union - this construction doesn't specify how to select which data representation is valid at the moment. In some cases ANY representation is valid, but many developers use that to save data space and store different params which are not needed together. In this case the developer typically uses some additional field outside of union or a similar mechanism to identify what union field is needed to use. Deep functions won't work properly without this information which must be provided explicitly.
  2. Flexible array (or array for which length isn't set) - typically it's the last field of structure - arrays for which the element count isn't set. For multidimensional arrays it's only allowed not to set the top array level length. The information about the actual length is typically stored outside of the array and C doesn't define a standard way to identify that source. Deep functions won't work properly in case this information isn't provided. There may be some compiler specific extensions, e.g. counted_by in clang. Unfortunately this field attribute isn't part of gcc and also isn't available as debug information which comes from clang.
  3. Pointers - C language allows to reference a single object OR array of the objects using pointer syntax. We assume typically that if the structure where the pointer is located doesn't have the size of the len field - we're dealing with 1 object. There is one more case - null-ended strings. C doesn't require to specify this. the situation gets even more complex when we use pointers of pointers and etc. Deep functions must have information about what particular scenario is necessary to use. In contrast - in Golang pointer always means 1 object and it has a special type - string.
  4. void* case, or similar when the cast one pointer type to another. Deep functions must know information about potential casting. E.g. in Linux kernel it's one of the common practices to have a void * private_data field in the common generic objects. In Go this issue is solved with interfaces which has a type id of the object stored in the pointer.
  5. As an extension of the previous case offset_of/continer_of, when a pointer points to the part of the structure and this macro is used to restore information about the container type based on some external data. The standard Linux kernel single/double-linked lists are built using this methodology. Go prohibits tricks like this because it doesn't allow arithmetic with pointers.

The summary: ambiguities are created by unions, flexible arrays and several extended use-cases of pointers. Those are well known flaws of C and the common practice is - to add attributes like clang added counted_by in the language itself, or in some auxiliary locations, e.g. MS RPC IDL defines attributes (this is just for reference):

  1. size_is - similar to counted_by, but also works with pointers. There is a good note about multidimential arrays and their difference from multidimential pointers
  2. string which informs that the char array or pointer actually is pointing to the null-terminated string
  3. switch_is for selecting the current union field

Names are not standardized, but the idea is similar. Another similar idea can be borrowed from Golang. It has tags which can be set for every structure field. We may want to support the same, since we know that tags are used in yaml/json and other modules and provide guidance on what to do with the field.

Potentially we could use only 'text-based tags'. But it is a bit too cumbersome for now to parse those parameters. It was decided to have support of text based tags AND callback handlers with some arbitrary data. The feature is called metac_tag_map_t and its API can be found in metac/reflect/entry_tag.h. In contrast with Golang, Metac doesn't taint type information with tag. Instead it allows you to create a hashmap or even a hierarchy of hashmaps using metac_entry_t as a key and metac_entry_tag_t as value.

This file also contains declarations of text-based tag functions and helper macroses:

metac_name_t metac_entry_tag_string_lookup(metac_entry_tag_t *p_tag, metac_name_t in_key);

#define METAC_TAG_STRING(_string_) ...
#define METAC_TAG_QSTRING(_string_...) ...

Pay attention to the latest one - it allows not to quote strings, but works similar to Golang backward quotes strings.

And the last - this file declares non-string based tags for the ambiguity cases we shared above and we'll demonstrate how to use them very soon:

#define METAC_COUNT_BY(_fld_) ...
#define METAC_ZERO_ENDED_STRING() ...
#define METAC_CONTAINER_OF(_container_type_, _member_) ...
#define METAC_UNION_MEMBER_SELECT_BY(_fld_, _cases_...) ...
// goes with optional
struct metac_union_member_select_by_case {
    metac_num_t fld_val;
    metac_name_t union_fld_name;
    metac_num_t union_fld_id;   /* id is used when name is NULL*/
};

#define METAC_CAST_PTR(_count_fld_, _selector_fld_, _cases_...) ...
// goes with
struct metac_ptr_cast_case {
    metac_num_t fld_val;
    metac_entry_t * p_ptr_entry;    // new type
};
// and
#define METAC_ENTRY_OF(_type_) ...

'Deep' functions declarations:

char * metac_value_string_ex(metac_value_t * p_val, metac_value_walk_mode_t wmode, metac_tag_map_t * p_tag_map);
int metac_value_equal_ex(metac_value_t * p_val1, metac_value_t * p_val2, 
    metac_value_memory_map_mode_t mode,
    metac_tag_map_t * p_tag_map);
metac_value_t *metac_value_copy_ex(metac_value_t * p_src_val, metac_value_t * p_dst_val,
    metac_value_memory_map_mode_t mode,
    void *(*calloc_fn)(size_t nmemb, size_t size),
    void (*free_fn)(void *ptr),
    metac_tag_map_t * p_tag_map);
metac_flag_t metac_value_free_ex(metac_value_t * p_val,
    metac_value_memory_map_non_handled_mode_t mode,
    void (*free_fn)(void *ptr),
    metac_tag_map_t * p_tag_map);

we also define:

static inline char * metac_value_string(metac_value_t * p_val) {
    return (p_val != NULL)?metac_value_string_ex(p_val, METAC_WMODE_shallow, NULL):NULL;
}

but this is no big value in shorter versions. We will cover here the version with _ex that may mean extended or extensive. ALL of them accept metac_tag_map_t * p_tag_map and use that parameter to get guidance on how to treat data in some cases. Typically there is also mode argument that allows to fine-tune the way how the particular function works and some memory related callbacks for the functions which will work with dynamic memory. Standard calloc and free will fit for those callbacks, but it may be useful to have your own callback. We're also using them for debug/unit-test purposes to make sure that we're getting the correct number of allocation and frees. One last note: deep functions are written in a way so they don't use stack recursion. C has a limited stack site and in case of very deep complex data that could be an issue if we used stack. Instead there is a special iterator which uses dynamic memory instead.

Let's write a series of unit-tests for our Demo application which will use 'deep' functions in step_05.

Here is how to print p_db: we need to adjust the test:

METAC_START_TEST(append_test) {
    struct {
        person_t * p_in;
        int expected_err;
        char * expected_string; // << added
    ...
        fail_unless((res != 0) == (tcs[tc_inx].expected_err != 0), "unexpected err result %i, expected %i", 
            res, tcs[tc_inx].expected_err);
        // << added from here
        if (tcs[tc_inx].expected_err == 0 && res == 0) {
            // let's print the data
            char * res_str;
            metac_value_t *p_db_value = METAC_VALUE_FROM_DECLLOC(loc, p_db);
            fail_unless(p_db_value != NULL);

            res_str = metac_value_string(p_db_value);
            fail_unless(res_str != NULL, "tc_inx %i: string is NULL", tc_inx);

            fail_unless(tcs[tc_inx].expected_string != NULL &&
                strcmp(res_str, tcs[tc_inx].expected_string) == 0, "tc_inx %i: got %s, expected %s",
                tc_inx, res_str, tcs[tc_inx].expected_string);

            free(res_str);
            metac_value_delete(p_db_value);
        }

If we run this now we'll see:

0%: Checks: 1, Failures: 1, Errors: 0
/home/test/metac/doc/demo/step_05/demodb_test.c:61:F:default:append_test:0: tc_inx 1: got 0x15b705ab0, expected (null)
make[1]: *** [bin_test] Error 1

value print prints value as address. This is right, but we probably want to see the DB internals. We need to switch to metac_value_string_ex(p_db_value, METAC_WMODE_deep, NULL);. We after that we got

0%: Checks: 1, Failures: 1, Errors: 0
/home/test/metac/doc/demo/step_05/demodb_test.c:61:F:default:append_test:0: tc_inx 1: got (db_t []){{.count = 1, .data = {},},}, expected (null)

better, but .data isn't shown. This is because it's a flexible array. We need to create a tag_map.

#include "demodb.c"

// define a tag_map for demodb.c << added starting from here
METAC_TAG_MAP_NEW(new_demodb_tag_map, NULL, {.mask = 
            METAC_TAG_MAP_ENTRY_CATEGORY_MASK(METAC_TEC_member)},)

    METAC_TAG_MAP_ENTRY_FROM_TYPE(db_t)
        METAC_TAG_MAP_SET_TAG(0, METAC_TEO_entry, 0, METAC_TAG_MAP_ENTRY_MEMBER({.n="data"}),
            METAC_COUNT_BY(count)
        )
    METAC_TAG_MAP_ENTRY_END

METAC_TAG_MAP_END
...
    };
    metac_tag_map_t * p_tag_map = new_demodb_tag_map(); // << added creation of our tagmap
    WITH_METAC_DECLLOC(loc, db_t * p_db = new_db());
...
            res_str = metac_value_string_ex(p_db_value, METAC_WMODE_deep, p_tag_map); // << added p_tag_map as last arg
            fail_unless(res_str != NULL, "tc_inx %i: string is NULL", tc_inx);
...
    db_delete(p_db);
    metac_tag_map_delete(p_tag_map); // << don't forget to clean up tag_map
}END_TEST

Try to run:

0%: Checks: 1, Failures: 1, Errors: 0
/home/test/metac/doc/demo/step_05/demodb_test.c:79:F:default:append_test:0: tc_inx 1: got (db_t []){{.count = 1, .data = {{.firstname = (char []){'J',}, .lastname = (char []){'D',}, .age = 43, .marital_status = msMarried,},},},}, expected (null)

Better, but char* is printed as a pointer to 1 char. We need to update tag_map with additional information. Updated tag_map:

// define a tag_map for demodb.c
METAC_TAG_MAP_NEW(new_demodb_tag_map, NULL, {.mask = 
            METAC_TAG_MAP_ENTRY_CATEGORY_MASK(METAC_TEC_member)},)

    METAC_TAG_MAP_ENTRY_FROM_TYPE(db_t)
        METAC_TAG_MAP_SET_TAG(0, METAC_TEO_entry, 0, METAC_TAG_MAP_ENTRY_MEMBER({.n="data"}),
            METAC_COUNT_BY(count)
        )
    METAC_TAG_MAP_ENTRY_END

    METAC_TAG_MAP_ENTRY_FROM_TYPE(person_t) // << added section: +7 next lines 
        METAC_TAG_MAP_SET_TAG(0, METAC_TEO_entry, 0, METAC_TAG_MAP_ENTRY_MEMBER({.n="firstname"}),
            METAC_ZERO_ENDED_STRING()
        )
        METAC_TAG_MAP_SET_TAG(0, METAC_TEO_entry, 0, METAC_TAG_MAP_ENTRY_MEMBER({.n="lastname"}),
            METAC_ZERO_ENDED_STRING()
        )
    METAC_TAG_MAP_ENTRY_END

METAC_TAG_MAP_END

The result is:

0%: Checks: 1, Failures: 1, Errors: 0
/home/test/metac/doc/demo/step_05/demodb_test.c:84:F:default:append_test:0: tc_inx 1: got (db_t []){{.count = 1, .data = {{.firstname = "Joe", .lastname = "Doe", .age = 43, .marital_status = msMarried,},},},}, expected (null)

That now works as expected. We just need to add expected_string value for each case:

...
        {
            .p_in = (person_t[]){{
                .firstname="Joe",
                .lastname="Doe",
                .age = 43,
                .marital_status = msMarried,
            }},
            .expected_err = 0,
            .expected_string = "(db_t []){{.count = 1, .data = {{.firstname = \"Joe\", .lastname = \"Doe\", .age = 43, .marital_status = msMarried,},},},}",
        },
        {
            .p_in = (person_t[]){{
                .firstname="Jane",
                .lastname="Doe",
                .age = 34,
                .marital_status = msMarried,
            }},
            .expected_err = 0,
            .expected_string = "(db_t []){{.count = 2, .data = {"
                "{.firstname = \"Joe\", .lastname = \"Doe\", .age = 43, .marital_status = msMarried,}, "
                "{.firstname = \"Jane\", .lastname = \"Doe\", .age = 34, .marital_status = msMarried,},"
            "},},}",
        },
        {
            .p_in = (person_t[]){{
                .firstname="Jack",
                .lastname="Doe",
                .age = 3,
                .marital_status = msSingle,
            }},
            .expected_err = 0,
            .expected_string = "(db_t []){{.count = 3, .data = {"
                "{.firstname = \"Joe\", .lastname = \"Doe\", .age = 43, .marital_status = msMarried,}, "
                "{.firstname = \"Jane\", .lastname = \"Doe\", .age = 34, .marital_status = msMarried,}, "
                "{.firstname = \"Jack\", .lastname = \"Doe\", .age = 3, .marital_status = msSingle,},"
            "},},}",
        },
...

And it passes ok:

100%: Checks: 1, Failures: 0, Errors: 0

There are some limitations metac_value_string_ex has:

  1. it doesn't know how to handle container_of case, because it's not clear how to print it out.
  2. it will fail in case of loop in structure of pointers, because it's not clear how to handle that. There is one idea, but for now this function just returns NULL.

The rest of the 'deep' functions handle such cases and don't fail.

The rest of tag types are covered with unit-tests - please refer to them to get some ideas how to use them.

Let's cover some aspects of 'deep'-copy/equal/free functions. All of the functions accept parameter mode:

/* default behavior in case callback didn't return information */
typedef enum {
    METAC_UPTR_fail = 0,    /* fail if met void* or ptr to declaration type (without knowing of internal structure) - safest and default*/
    METAC_UPTR_ignore,      /* mark them as ignored - don't do anything with them. Set them as NULL during copying, don't compare during comparison */
    METAC_UPTR_as_values,   /* treat them as long. copy them as unsigned long values on copy/comparison (ptr without children). delete will ignore it.*/
    METAC_UPTR_as_one_byte_ptr, /* the size isn't known, but for sure it's at least 1 byte. This is done more for free memory. */
}metac_value_memory_map_uptr_mode_t;
typedef enum {
    METAC_UNION_fail = 0,    /* fail if met union - safest and default*/
    METAC_UNION_ignore,      /* mark as ignored - will set to 0, don't compare on equal. Warning, if union contains pointers - there can be leaks*/
    METAC_UNION_as_mem,      /* use memcmp/memcpy to treat. Warning, if union contains pointers - there can be leaks*/
}metac_value_memory_map_union_mode_t;
typedef enum {
    METAC_FLXARR_fail = 0,    /* fail if met flexible array (array with not set length) - safest and default*/
    METAC_FLXARR_ignore,      /* mark as ignored - won't do anything. Warning, if array elements contains pointers - there can be leaks*/
}metac_value_memory_map_flex_array_mode_t;
typedef struct {
    metac_value_memory_map_uptr_mode_t unknown_ptr_mode;
    metac_value_memory_map_union_mode_t union_mode;
    metac_value_memory_map_flex_array_mode_t flex_array_mode;
}metac_value_memory_map_non_handled_mode_t;
typedef struct {
    enum {
        METAC_MMODE_dag = 0,    /* analyse pointers to detect map structure (can be DAG with weak pointers) -safest and default */
        METAC_MMODE_tree,       /* treat every pointer as a separate mem block */
    }memory_block_mode;
    metac_value_memory_map_uptr_mode_t unknown_ptr_mode;
    metac_value_memory_map_union_mode_t union_mode;
    metac_value_memory_map_flex_array_mode_t flex_array_mode;
}metac_value_memory_map_mode_t;

In all cases the safest option goes as the first one. There is a big text written in the beginning of value_deep.c that covers the implementation details. DAG mode is default and in this case the code identifies all actually allocated blocks. If there are some weak pointers, which point to some field of the block, the code treats it as weak. In case of TREE mode we treat all pointers as non-weak and allocate memory blocks even though originally the pointer was pointing in the middle of another structure the implementation will create a copy of that field. We think that for some cases it's a nice feature - just get rid of weak pointers by copying in TREE mode. This parameter is also needed for 'deep'-equal function. It's possible in TREE mode to compare the original complex data-structure with weak pointers with its copy made in TREE mode. In other words TREE mode will be more tolerant to the differences in structure made by the pointer. But it will compare all values and resulting tree structure.

'Deep'-free has to be precise on what memory blocks to free, that's why it doesn't have memory_block_mode parameter and always works in DAG mode.

We can try to work with all of them in the next unit-test:

METAC_START_TEST(deep_test) {
    metac_tag_map_t * p_tag_map = new_demodb_tag_map();
    WITH_METAC_DECLLOC(loc, 
        db_t * p_db = new_db(), *p_db_backup = NULL);
    fail_unless(p_tag_map != NULL, "new_demodb_tag_map failed");

    fail_unless(db_append(&p_db, (person_t[]){{
		.firstname="Joe",
		.lastname="Doe",
		.age = 43,
		.marital_status = msMarried,
	}}) == 0, "db_append 1 failed");

    /* create values */
    metac_value_t *p_db_value = METAC_VALUE_FROM_DECLLOC(loc, p_db);
    fail_unless(p_db_value != NULL);
    metac_value_t *p_db_backup_value = METAC_VALUE_FROM_DECLLOC(loc, p_db_backup);
    fail_unless(p_db_backup_value != NULL);

    metac_value_memory_map_mode_t mode = {.memory_block_mode = METAC_MMODE_dag,};

    fail_unless(metac_value_copy_ex(p_db_value, p_db_backup_value,
        &mode, calloc, free, p_tag_map) != NULL, "copy failed");

    char * str = NULL, * expected_str = NULL;

    str = metac_value_string_ex(p_db_backup_value, METAC_WMODE_deep, p_tag_map);
    expected_str = "(db_t []){{.count = 1, .data = {{.firstname = \"Joe\", .lastname = \"Doe\", .age = 43, .marital_status = msMarried,},},},}";

    fail_unless(str != NULL && strcmp(str, expected_str) == 0, "got %s, expected %s", str, expected_str);
    free(str);

    fail_unless(metac_value_equal_ex(p_db_value, p_db_backup_value,
        &mode, p_tag_map) == 1, "expected db and backup be equal");

    // we even can use db api for copy
    fail_unless(db_append(&p_db_backup, (person_t[]){{
		.firstname="Jane",
		.lastname="Doe",
		.age = 34,
		.marital_status = msMarried,
	}}) == 0, "db_append 1 failed");

    str = metac_value_string_ex(p_db_backup_value, METAC_WMODE_deep, p_tag_map);
    expected_str = "(db_t []){{.count = 2, .data = {"
        "{.firstname = \"Joe\", .lastname = \"Doe\", .age = 43, .marital_status = msMarried,}, "
        "{.firstname = \"Jane\", .lastname = \"Doe\", .age = 34, .marital_status = msMarried,},"
    "},},}";

    fail_unless(str != NULL && strcmp(str, expected_str) == 0, "got %s, expected %s", str, expected_str);
    free(str);

    fail_unless(metac_value_equal_ex(p_db_value, p_db_backup_value,
        &mode, p_tag_map) == 0, "expected db and backup be NOT equal");

    // and clean data with 'deep function' instead of db api
    metac_value_memory_map_non_handled_mode_t fmode = {0, };
    fail_unless(metac_value_free_ex(p_db_value, &fmode, free, p_tag_map) != 0, "wasn't able to free db"); 

    metac_value_delete(p_db_backup_value);
    metac_value_delete(p_db_value);

    db_delete(p_db_backup);
    metac_tag_map_delete(p_tag_map);
}END_TEST

This test passes successfully.

Work with function parameters

In contrast to the rest of this document we're going to use a separate example to demonstrate ability to parse parameters of functions.

Note: The same will work for function pointers.

The example will have a goal to print all parameters which the function accepts and result which the function returns (if any). In order to keep the list of arguments Metac introduces a special type called metac_parameter_storage_t. The internals of this type isn't exposed to the user, but it goes with the list of basic functions to create/copy/cleanup/delete:

/** @brief copy all internals of one metac_parameter_storage_t to another metac_parameter_storage_t */
int metac_parameter_storage_copy(metac_parameter_storage_t * p_src_param_storage, metac_parameter_storage_t * p_dst_param_storage);
/** @brief clean metac_parameter_storage_t all internal information, but preserve metac_parameter_storage_t */
void metac_parameter_storage_cleanup(metac_parameter_storage_t * p_param_storage);
/** @brief delete metac_parameter_storage_t (calls cleanup prior to deletion) */
void metac_parameter_storage_delete(metac_parameter_storage_t * p_param_load);
/** @brief get number of parameters in metac_parameter_storage_t */
metac_num_t metac_parameter_storage_size(metac_parameter_storage_t * p_param_load);

This set of functions is put into metac/base.h to emphasize that this is a foundation object type which is needed to store function parameters. The user should think about this as of analog of va_list type. Unfortunately it's not possible to use va_list to store list of arguments because of the following reasons: it doesn't keep its size, it's very fragile to work with and finally there is no possibility to create/modify va_list except by creating some hacks (e.g. see va_list_ex.h, but it is only used for testing purposes). metac_parameter_storage_t has some extra API to manipulate the parameters list and even a function which will create metac_value_t out of parameter:

/** @brief extra function to append the parameter storage with inner parameter storage.
 * useful for cases when you have va_list or unspecified parameter 
 */
int metac_parameter_storage_append_by_parameter_storage(metac_parameter_storage_t * p_param_storage,
    metac_entry_t *p_entry);
/** @brief extra function to append the parameter storage with the buffer to store date of the parameter 
 */
int metac_parameter_storage_append_by_buffer(metac_parameter_storage_t * p_param_storage,
    metac_entry_t *p_entry,
    metac_num_t size);

/** @brief extra function to convert n-th parameter of parameter storage to metac_value_t */
metac_value_t * metac_parameter_storage_new_param_value(metac_parameter_storage_t * p_param_storage, metac_num_t id);

Though it's not necssary to work with those functions directly in most cases. There are 2 additional functions which represent the high-level API and can be used to fill in metac_parameter_storage_t and even create a wrapping metac_value_t to be used in deep print/copy/compare/delete functions. For metac_value_t which kind is subprogram(that means it is a function) or subroutine (that means that it represents the type of pointer to a function) or va_list/unspecified( or ... ) parameter metac_parameter_storage_t * must be put as addr parameter. This is exactly what the following 2 function do:

/** @brief function will parse all parameters added as ... based on the parameter list given in p_entry (must be subprogram or subroutine), put them into p_subprog_load and will create a wrapping metac_value_t
   tag_map is needed in case the function in p_entry has unspecified parameter or va_list */
metac_value_t * metac_value_parameter_wrap(metac_value_t * p_val,metac_tag_map_t * p_tag_map, ...);
/** @brief function will parse all parameters added as `va_list parameters` based on the parameter list given in p_entry (must be subprogram or subroutine), put them into p_subprog_load and will create a wrapping metac_value_t
   tag_map is needed in case the function in p_entry has unspecified parameter or va_list */
metac_value_t * metac_value_parameter_vwrap(metac_value_t * p_val,metac_tag_map_t * p_tag_map, va_list parameters);

Let's create a small example to see how to use those 2 highlevel functions. Let's say we have a test function and main function that calls that function:

int test_function1_with_args(int a, short b){
    return a + b + 6;
}

int main() {
    printf("fn returned: %i\n", test_function1_with_args(10, 22));
    return 0;
}

We can store it's arguments into a single metac_value_t if we modify this code like this:

#include "metac/reflect.h"
#include <stdlib.h> /*free*/

#define METAC_WRAP_FN_RES(_tag_map_, _fn_, _args_...) ({ \
        metac_parameter_storage_t * p_param_storage = metac_new_parameter_storage(); \
        if (p_param_storage != NULL) { \
            p_val = metac_value_parameter_wrap(metac_new_value(METAC_GSYM_LINK_ENTRY(_fn_), p_param_storage), _tag_map_, _args_); \
        } \
        _fn_(_args_);\
})

int test_function1_with_args(int a, short b){
    return a + b + 6;
}
METAC_GSYM_LINK(test_function1_with_args);

int main() {
    metac_value_t * p_val = NULL;
    printf("fn returned: %i\n", METAC_WRAP_FN_RES(NULL, test_function1_with_args, 10, 22));
    if (p_val != NULL) {
        char * s = metac_value_string_ex(p_val, METAC_WMODE_deep, NULL);
        if (s != NULL) {
            printf("captured %s\n", s);
            free(s);
        }
        metac_parameter_storage_t * p_param_storage = (metac_parameter_storage_t *)metac_value_addr(p_val);
        metac_parameter_storage_delete(p_param_storage);
        metac_value_delete(p_val);
    }
    return 0;
}

If we run this the output will be:

step_06 % ./param_demo
fn returned: 38
captured test_function1_with_args(10, 22)

The code above along with Makefile is available in the folder step_06.

As we can see p_val contains information about function and its argument which were used. p_val was set by the macros METAC_WRAP_FN_RES which performs 3 things:

  1. create metac_parameter_storage_t
  2. put all parameters provided to macro into that metac_parameter_storage_t and wrap it by metac_value_t. All of this is done by metac_new_value_with_parameters function.
  3. call the _fn_ with the same parameters and return the result.

metac_new_value_with_parameters supports different types of parameters:

  1. base types (e.g. char, int, and etc)
  2. enums
  3. structs
  4. pointers (that includes arrays, because in C if the used puts array as parameter the function receives pointer to the first element)
  5. unspecified parameters (...) and va_lists. Due to the fact that there is no way to understand the numbrer of parameters inside sich parameters metac_value_parameter_wrap needs non-NULL parameter p_tag_map to be set in this case.

Let's update our code to show how to work with unspecified parameters:

...
int my_printf(const char * format, ...) {
    va_list l;
    va_start(l, format);
    int res = vprintf(format, l);
    va_end(l);
    return res;
}
METAC_GSYM_LINK(my_printf);

METAC_TAG_MAP_NEW(va_args_tag_map, NULL, {.mask = 
            METAC_TAG_MAP_ENTRY_CATEGORY_MASK(METAC_TEC_variable) |
            METAC_TAG_MAP_ENTRY_CATEGORY_MASK(METAC_TEC_func_parameter) | 
            METAC_TAG_MAP_ENTRY_CATEGORY_MASK(METAC_TEC_member) |
            METAC_TAG_MAP_ENTRY_CATEGORY_MASK(METAC_TEC_final),},)
    /* start tags for all types */

    METAC_TAG_MAP_ENTRY(METAC_GSYM_LINK_ENTRY(my_printf))
        METAC_TAG_MAP_SET_TAG(0, METAC_TEO_entry, 0, METAC_TAG_MAP_ENTRY_PARAMETER({.n = "format"}),
            METAC_ZERO_ENDED_STRING()
        )
        METAC_TAG_MAP_SET_TAG(0, METAC_TEO_entry, 0, METAC_TAG_MAP_ENTRY_PARAMETER({.i = 1}), 
            METAC_FORMAT_BASED_VA_ARG()
        )
    METAC_TAG_MAP_ENTRY_END

METAC_TAG_MAP_END


int main() {
    metac_tag_map_t * p_tagmap = va_args_tag_map();

    metac_value_t * p_val = NULL;
    printf("fn returned: %i\n", METAC_WRAP_FN_RES(p_tagmap, my_printf, "%d %d\n", 10, 22));
    if (p_val != NULL) {
        char * s = metac_value_string_ex(p_val, METAC_WMODE_deep, p_tagmap);
        if (s != NULL) {
            printf("captured %s\n", s);
            free(s);
        }
        metac_parameter_storage_t * p_param_storage = (metac_parameter_storage_t *)metac_value_addr(p_val);
        metac_parameter_storage_delete(p_param_storage);
        metac_value_delete(p_val);
    }

    metac_tag_map_delete(p_tagmap);
    return 0;
}

If we run this example available in the folder step_07, we'll see:

step_07 % ./param_demo
10 22
fn returned: 6
captured my_printf("%d %d\n", (int)10, (int)22)

We already familiar with tag_map concept. To tell that parameter ... is defined by the previous argument we used the folowing lines:

        METAC_TAG_MAP_SET_TAG(0, METAC_TEO_entry, 0, METAC_TAG_MAP_ENTRY_PARAMETER({.i = 1}), 
            METAC_FORMAT_BASED_VA_ARG()

The current implementation of METAC_FORMAT_BASED_VA_ARG is always to get the previous parameter and to treat it as printf-format string.

Important notes:

  1. metac_parameter_storage_t allocates only memory for the parameter itself. If the parameter is a pointer and the function changed the argument stored by that pointer it doesn't affect anyhow metac_parameter_storage_t. If we call metac_value_string_ex after the actual function call for the created by metac_value_parameter_wrap value - we'll see the updated value.
  2. If we want to keep the values of parameters, including all values of pointer parameters, we can use deep copy function. This may be consideres as a snapshot of the parameters values. For that purpose we must create another empty metac_parameter_storage_t, create metac_valut_t which will use that metac_parameter_storage_t as address and subprogram or subroutine metac_entry_t as entry. Below we can see the example taken from the test. Please note that to cleanup all values to which pointer parameters pointed it is necessary to use deep free function metac_value_free_ex.
METAC_START_TEST(args_deep_copy_and_delete_sanity) {
    metac_tag_map_t * p_tagmap = va_args_tag_map();
    metac_value_t * p_val1, *p_val2;
    metac_parameter_storage_t * p_param_storage;
    char *s, *expected_s;

    int * test_arr1 = (int[]){0, 1, 2, 3};
    p_val1 = METAC_NEW_VALUE_WITH_ARGS_FOR_FN(p_tagmap, test_array_len, test_arr1, 4);
    fail_unless(p_val1 != NULL);

    expected_s = "test_array_len((int []){0, 1, 2, 3,}, 4)";
    s = metac_value_string_ex(p_val1, METAC_WMODE_deep, p_tagmap);
    fail_unless(s != NULL);
    fail_unless(strcmp(s, expected_s) == 0, "got %s, expected %s", s, expected_s);
    free(s);

    p_param_storage = metac_new_parameter_storage();
    p_val2 = metac_new_value(METAC_GSYM_LINK_ENTRY(test_array_len), p_param_storage);

    fail_unless(metac_value_copy_ex(p_val1, p_val2, NULL, NULL, NULL, p_tagmap) == p_val2);

    s = metac_value_string_ex(p_val2, METAC_WMODE_deep, p_tagmap);
    fail_unless(s != NULL);
    fail_unless(strcmp(s, expected_s) == 0, "got %s, expected %s", s, expected_s);
    free(s);

    test_arr1[0] = 1;

    // the ideat is that the change in the p_val args won't affect the string, because all artguments are copied with deep-copy
    s = metac_value_string_ex(p_val2, METAC_WMODE_deep, p_tagmap);
    fail_unless(s != NULL);
    fail_unless(strcmp(s, expected_s) == 0, "got %s, expected %s", s, expected_s);
    free(s);

    // we need to cleanup allocated by deep copy memory
    fail_unless(metac_value_free_ex(p_val2, NULL, NULL, p_tagmap) == 1);

    METAC_VALUE_WITH_ARGS_DELETE(p_val2);

    METAC_VALUE_WITH_ARGS_DELETE(p_val1);

    metac_tag_map_delete(p_tagmap);
}END_TEST
  1. The demonstrated method can be used to identify if arguments were changed during the call. For this purpose it will be necessary to create a value with the arguments and make a deep copy of it before the call. After the call the first value will contain the modified values and the second value will contain the snapshotted prior to the call values. If deep equial function shows the difference - the parameters were modified during the call.

More examples on the function parameters can be found here and in the tests. That includes all types of arguments and work with pointers of functions.

We were able to go through the main concepts, caveates and solutions offered by Metac. Thanks for reading!