Multiple errors when trying to use ctypeslib2 instead of ctypeslib

[FloSim82]

Hello, i'm facing different issues when trying to use ctypeslib2 instead of ctypeslib.
I listed them all above, and I'm sorry to use one ticket but I don't know if they are related...

1- Python2 installation issue

First, what I did is :

sudo apt remove python-ctypeslib
pip install --user ctypeslib2

But I'm facing this issue :

Looking in indexes: https://pypi.org/simple, http://komodo.pfa.tds:7070
Collecting ctypeslib2
  Using cached ctypeslib2-2.3.2.tar.gz (74 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 331, in run
    resolver.resolve(requirement_set)
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 516, in prepare_linked_requirement
    req, self.req_tracker, self.finder, self.build_isolation,
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_internal/distributions/sdist.py", line 33, in prepare_distribution_metadata
    self.req.load_pyproject_toml()
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 516, in load_pyproject_toml
    str(self)
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_internal/pyproject.py", line 75, in load_pyproject_toml
    pp_toml = pytoml.load(f)
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_vendor/pytoml/parser.py", line 11, in load
    return loads(fin.read(), translate=translate, object_pairs_hook=object_pairs_hook, filename=getattr(fin, 'name', repr(fin)))
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_vendor/pytoml/parser.py", line 24, in loads
    ast = _p_toml(src, object_pairs_hook=object_pairs_hook)
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_vendor/pytoml/parser.py", line 341, in _p_toml
    s.expect_eof()
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_vendor/pytoml/parser.py", line 123, in expect_eof
    return self._expect(self.consume_eof())
  File "/home/fsimon/.local/lib/python2.7/site-packages/pip/_vendor/pytoml/parser.py", line 163, in _expect
    raise TomlError('msg', self._pos[0], self._pos[1], self._filename)
TomlError: /tmp/pip-install-PNSRSy/ctypeslib2/pyproject.toml(31, 1): msg

2- Python2 Import issue

I tried on a Ubuntu 18.04 machine and it installed successfully the 2.3.2 version.
But when I try to import ctypeslib in python2 console, I see this issue :

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/default/.local/lib/python2.7/site-packages/ctypeslib/__init__.py", line 48, in <module>
    from ctypeslib.codegen.codegenerator import translate, translate_files
  File "/home/default/.local/lib/python2.7/site-packages/ctypeslib/codegen/codegenerator.py", line 1008
    def __init__(self, cfg: config.CodegenConfig):
                          ^
SyntaxError: invalid syntax

3- Python3 Import issue

Thus, I also tried with python3, it installed successfully the 2.3.2 version.
But when I try to import ctypeslib in python3 console, or when launching clang2py, I see this issue :

Traceback (most recent call last):
  File "/home/fsimon/.local/bin/clang2py", line 5, in <module>
    from ctypeslib.clang2py import main
  File "/home/fsimon/.local/lib/python3.5/site-packages/ctypeslib/__init__.py", line 48, in <module>
    from ctypeslib.codegen.codegenerator import translate, translate_files
  File "/home/fsimon/.local/lib/python3.5/site-packages/ctypeslib/codegen/codegenerator.py", line 21, in <module>
    from ctypeslib.codegen import config
  File "/home/fsimon/.local/lib/python3.5/site-packages/ctypeslib/codegen/config.py", line 9
    symbols: list = []
           ^
SyntaxError: invalid syntax

Any ideas? Thanks!

[trolldbois]

All 3 issues are due to python version. The code is python 3.8 and later.

[FloSim82]

Thanks for your answer, indeed it works with python 3.8.
I did not find any information concerning this requirement, can you ideally please some info in your home page?
Thanks

[FloSim82]

One more question, once again related to Ubuntu distributions.
I took the basic "t.c"' example found in https://pypi.org/project/ctypeslib2/.

1- launch clang2py on Ubuntu 16.04

~$ clang2py t.c --debug

returns nothing, whereas it should return an output. Is it normal?

2- launch clang2py on Ubuntu 18.04

~$ clang2py t.c --debug

returns the following, which is not the same exact expected output. Is it normal?

# -*- coding: utf-8 -*-
#
# TARGET arch is: []
# WORD_SIZE is: 8
# POINTER_SIZE is: 8
# LONGDOUBLE_SIZE is: 16
#
import ctypes


class AsDictMixin:
    @classmethod
    def as_dict(cls, self):
        result = {}
        if not isinstance(self, AsDictMixin):
            # not a structure, assume it's already a python object
            return self
        if not hasattr(cls, "_fields_"):
            return result
        # sys.version_info >= (3, 5)
        # for (field, *_) in cls._fields_:  # noqa
        for field_tuple in cls._fields_:  # noqa
            field = field_tuple[0]
            if field.startswith('PADDING_'):
                continue
            value = getattr(self, field)
            type_ = type(value)
            if hasattr(value, "_length_") and hasattr(value, "_type_"):
                # array
                if not hasattr(type_, "as_dict"):
                    value = [v for v in value]
                else:
                    type_ = type_._type_
                    value = [type_.as_dict(v) for v in value]
            elif hasattr(value, "contents") and hasattr(value, "_type_"):
                # pointer
                try:
                    if not hasattr(type_, "as_dict"):
                        value = value.contents
                    else:
                        type_ = type_._type_
                        value = type_.as_dict(value.contents)
                except ValueError:
                    # nullptr
                    value = None
            elif isinstance(value, AsDictMixin):
                # other structure
                value = type_.as_dict(value)
            result[field] = value
        return result


class Structure(ctypes.Structure, AsDictMixin):

    def __init__(self, *args, **kwds):
        # We don't want to use positional arguments fill PADDING_* fields

        args = dict(zip(self.__class__._field_names_(), args))
        args.update(kwds)
        super(Structure, self).__init__(**args)

    @classmethod
    def _field_names_(cls):
        if hasattr(cls, '_fields_'):
            return (f[0] for f in cls._fields_ if not f[0].startswith('PADDING'))
        else:
            return ()

    @classmethod
    def get_type(cls, field):
        for f in cls._fields_:
            if f[0] == field:
                return f[1]
        return None

    @classmethod
    def bind(cls, bound_fields):
        fields = {}
        for name, type_ in cls._fields_:
            if hasattr(type_, "restype"):
                if name in bound_fields:
                    if bound_fields[name] is None:
                        fields[name] = type_()
                    else:
                        # use a closure to capture the callback from the loop scope
                        fields[name] = (
                            type_((lambda callback: lambda *args: callback(*args))(
                                bound_fields[name]))
                        )
                    del bound_fields[name]
                else:
                    # default callback implementation (does nothing)
                    try:
                        default_ = type_(0).restype().value
                    except TypeError:
                        default_ = None
                    fields[name] = type_((
                        lambda default_: lambda *args: default_)(default_))
            else:
                # not a callback function, use default initialization
                if name in bound_fields:
                    fields[name] = bound_fields[name]
                    del bound_fields[name]
                else:
                    fields[name] = type_()
        if len(bound_fields) != 0:
            raise ValueError(
                "Cannot bind the following unknown callback(s) {}.{}".format(
                    cls.__name__, bound_fields.keys()
            ))
        return cls(**fields)


class Union(ctypes.Union, AsDictMixin):
    pass





class struct_my_bitfield(Structure):
    pass

struct_my_bitfield._pack_ = 1 # source:False
struct_my_bitfield._fields_ = [
    ('a', ctypes.c_int64, 3),
    ('b', ctypes.c_int64, 4),
    ('c', ctypes.c_int64, 3),
    ('d', ctypes.c_int64, 3),
    ('f', ctypes.c_int64, 2),
    ('PADDING_0', ctypes.c_uint64, 49),
]

__all__ = \
    ['struct_my_bitfield']

Thanks.

[FloSim82]

1- launch clang2py on Ubuntu 16.04

~$ clang2py t.c --debug

returns nothing, whereas it should return an output. Is it normal?

I was able to have an output under Ubuntu 16.04 after installing LLVM 12 :)
I also installed LLVM 13 on Ubuntu 18.04 machine.
Hence, we have the same output on both (the "more complete one than on your doc", see above).

---

Now : I have DEBUG:cursorhandler:cursor.type:INVALID 'errors' leading to an empty output when parsing all files from a same directory (works with some files from here, but when adding other files I have this issue).

I think I'm not far from the solution, but do you know how could I resolve this issue?
Do you need the complete log, or all H files?

Thank you

[FloSim82]

After some more analysis, it appears that the following #define is making the issue happen (that leads to an empty output, as a reminder) :

void    log_print(const char * module, int options, int severity, const char * color, int output, const char * fmt, ...);
#define log_print_def(fmt, ...)     log_print(NULL, 0, DEBUG_INFO, NULL, LOG_DEF_PRINT, fmt, ##__VA_ARGS__)

For info, in debug logs:

DEBUG:clangparser:/usr/include/dt-sdk/lib_logs.h:158: Found a MACRO_DEFINITION|log_print_def|log_print_def
DEBUG:handler:get_unique_name: name "log_print_def"
DEBUG:utils:MACRO_DEFINITION: displayname:'log_print_def'
DEBUG:handler:get_unique_name: name "log_print_def"
DEBUG:handler:get_unique_name: name "log_print_def"
DEBUG:utils:_literal_handling: displayname:'log_print_def'
DEBUG:cursorhandler:literal has 23 tokens.[ log_print_def ( fmt , ... ) log_print ( NULL , 0 , DEBUG_INFO , NULL , LOG_DEF_PRINT , fmt , ## __VA_ARGS__ ) ]
DEBUG:cursorhandler:cursor.type:INVALID
DEBUG:cursorhandler:token:log_print_def tk.kd: IDENTIFIER tk.cursor.kd:MACRO_DEFINITION cursor.kd:MACRO_DEFINITION

Do you know why this happen?
Thanks again, and I apologize for my previous messages.

[FloSim82]

@trolldbois hello, any idea for this issue with this variadic arguments function ?

[FloSim82]

Hi, still anybody has a clue to avoid this issue?

[FloSim82]

Hello, I would like to know if maintenance is still ongoing on this library, because I still have the issue with #define based on a variadic function? Thanks

[trolldbois]

@FloSim82 Would you be able to open an issue, just for that, with a simple poc and the specific versions of python, clang and python clang package you are using ?
Does not seem to be a problem in my tests.

[FloSim82]

Thanks for your answer.
For the moment, after cleaning the libs and resintalling from scratch, I don't have any error anymore.
If encountering again the variadic macro issue, I'll report a new bug.

[FloSim82]

Issue #106 has bee n reported for the latest issue, thanks again