Iyfjky

Question

In the mathematical sense, a set (or type) is closed under an operation if the operation always returns a member of the set itself.

This question is about making a class that is closed under all operations inherited from its superclasses.

Consider the following class.

class MyInt(int):
 pass

Since __add__ has not been overridden, it is not closed under addition.

x = MyInt(6)
print(type(x + x)) # <class 'int'>

One very tedious way to make the type closed would be to manually cast back the result of every operation that returns an int to MyInt.

Here, I automated that process using a metaclass, but this seems like an overly complex solution.

import functools

class ClosedMeta(type):
 _register = 

 def __new__(cls, name, bases, namespace):
 # A unique id for the class
 uid = max(cls._register) + 1 if cls._register else 0

 def tail_cast(f):
 @functools.wraps(f)
 def wrapper(*args, **kwargs):
 out = f(*args, **kwargs)
 if type(out) in bases:
 # Since the class does not exist yet, we will recover it later
 return cls._register[uid](out)
 return out
 return wrapper

 for base in reversed(bases):
 for name, attr in base.__dict__.items():
 if callable(attr) and name not in namespace:
 namespace[name] = tail_cast(attr)

 subcls = super().__new__(cls, name, bases, namespace)
 cls._register[uid] = subcls
 return subcls

class ClosedInt(int, metaclass=ClosedMeta):
 pass

This fails on some cornercases such as property and methods recovered through __getattribute__. It also fails when the base is not composed only of base types.

By example, this fails:

class MyInt(int):
 pass

class ClosedInt(MyInt, metaclass=ClosedMeta):
 pass

ClosedInt(1) + ClosedInt(1) # returns the int 2

I attempted to fix this, but it just seems to go deeper and deeper in the rabbit hole.

This seems like a problem that might have some simple pythonic solution. What would be a other, neater ways to achieve such a closed type?

So, this mgiht be nitpicky, but class BasicIntContainer(int) is not an int container, it is an int. — 3 hours ago
@juanpa.arrivillaga I might be off in my terminology, how would you call such a construct? — 3 hours ago
It is simply inheritance, an "is-a" relationship in the parlance of OOP. If you hover over the containers tag, you'll see that: "A container is a class, a data structure, or an abstract data type whose instances are collections of other objects.". So like a list or a dict. Here, a BasicIntContainer instance isn't a collection of other objects, it is a specialized kind of int object. I don't want to distract from an interesting question with more terminology nitpicking. Anyway, I would just rephrase as "how to create a type that is closed under inherited operations" — 3 hours ago
You can easily write a simple decorator if you just want to cast return values to type itself, but the problem with that would be that there may be methods that wouldn't be returning values that would be able to cast to the proper type, if that is the case; you would need to distinguish them somehow, if not you can just decorate the class with a decorator that would decorate every method. — 3 hours ago
@IÃ…ÂŸÃ„Â±kKaplan For simplification, let's assume that the __new__ knows how to cast from all of its superclasses. Also, this decorator solution is basicaly what I implemented with a metaclass. I am currently fixing it, but it is getting very complex and I am for sure forgetting a lot of cornercases. — 3 hours ago

score 2 · Answer 1 · 2018-09-15 06:34:52Z

I think using a class decorator with a black list of methods that should not return objects of the same type would be somewhat more Pythonic:

class containerize:
 def __call__(self, obj):
 if isinstance(obj, type):
 return self.decorate_class(obj)
 return self.decorate_callable(obj)

 def decorate_class(self, cls):
 for name in dir(cls):
 attr = getattr(cls, name)
 if callable(attr) and name not in ('__class__', '__init__', '__new__', '__str__', '__repr__', '__getattribute__'):
 setattr(cls, name, self.decorate_callable(attr))
 return cls

 def decorate_callable(self, func):
 def wrapper(obj, *args, **kwargs):
 return obj.__class__(func(obj, *args, **kwargs))
 return wrapper

so that:

class MyInt(int):
 pass

@containerize()
class ClosedIntContainer(MyInt):
 pass

i = ClosedIntContainer(3) + ClosedIntContainer(2)
print(i, type(i))

would output:

5 <class '__main__.ClosedIntContainer'>

score 1 · Answer 2 · 2018-09-15 04:53:37Z

I still feel there might be a more natural way to accomplish this, but I was able to fix the attempt provided in the question.

Here are the main points that needed ot be fixed.

We must check methods of all classes in the mro, not only the bases;

__getattribute__ and __getattr__ must be treated as special cases;

Attributes with __get__ must be treated separately;

We have to write a list of exceptions as methods such as __int__ or __eq__ obviously should return their expected types.

Code

import functools

def get_mro(bases):
 # We omit 'object' as it is the base type
 return type('', bases, ).__mro__[1:-1]

class ClosedMeta(type):
 _register = 

 # Some methods return type must not change
 _exceptions = ('__int__', '__eq__', ...)

 def __new__(cls, name, bases, namespace):
 # A unique id for the class
 uid = max(cls._register) + 1 if cls._register else 0
 mro = get_mro(bases)

 def tail_cast(f):
 """Cast the return value of f"""
 @functools.wraps(f)
 def wrapper(*args, **kwargs):
 out = f(*args, **kwargs)
 if type(out) in mro:
 # Since the class does not exist yet, we will recover it later
 return cls._register[uid](out)
 return out
 return wrapper

 def deep_tail_cast(f):
 """Cast the return value of f or the return value of f(...)"""
 @functools.wraps(f)
 def wrapper(*args, **kwargs):
 out = f(*args, **kwargs)
 if callable(out):
 return tail_cast(out)
 elif type(out) in mro:
 return cls._register[uid](out)
 else:
 return out
 return wrapper

 class PropertyCast:
 """Cast the return value of a property"""
 def __init__(self, prop):
 self.prop = prop

 def __get__(self, instance, owner):
 return cls._register[uid](self.prop.__get__(instance, owner))

 def __set__(self, instance, value):
 return self.prop.__set__(instance, value)

 def __delete__(self, instance):
 return self.prop.__delete__(instance)

 for base in reversed(mro):
 for name, attr in base.__dict__.items():
 if name in ('__getattr__', '__getattribute__'):
 namespace[name] = deep_tail_cast(attr)
 elif callable(attr) and name not in namespace and name not in cls._exceptions:
 namespace[name] = tail_cast(attr)
 elif hasattr(attr, '__get__'):
 namespace[name] = PropertyCast(attr)

 subcls = super().__new__(cls, name, bases, namespace)
 cls._register[uid] = subcls
 return subcls

Example

class MyInt(int):
 def __getattr__(self, _):
 return 1

 @property
 def foo(self):
 return 2

class ClosedInt(MyInt, metaclass=ClosedMeta):
 pass

x = ClosedInt(2)
print(type(x * x), x * x)
print(type(x.foo), x.foo)
print(type(x.bar), x.bar)

Output

<class '__main__.ClosedIntContainer'> 4
<class '__main__.ClosedIntContainer'> 2
<class '__main__.ClosedIntContainer'> 1

This still has some issues. By example we still have the tedious task to go through all dunder methods an mark down exceptions to the implemented rules, but unless there is a list of those somewhere, this seems unavoidable.

score 0 · Answer 3 · 2018-09-15 04:32:54Z

This cannot be done, the data model forbids it. And I can prove it to you:

>>> class MyClass(ClosedInt, metaclass=type):
... def __add__(self, other):
... return 'potato'
... 
>>> MyClass(1) + ClosedInt(2)
'potato'

Addition is handled by the left hand object first, and if the left type handles it (i.e. does not return NotImplemented singleton) then nothing about other is considered in this operation. If the right hand type is a subclass of the left-hand type, you could control the result with the reflected method __radd__ - but of course that is impossible in the general case.

That would be fine since by definition, closure means if x and y are of type T, then x + y is of type T. Here MyClass() is not of type ClosedInt. So the question really applies only if both elements of the operation are of the ClosedInt type — 2 hours ago
Also can not be done, in general case, and the reason is the same - subclasses can override any behavior of parent. — 2 hours ago
@wim. See my attempt. It's not perfect, and of course you can bypass it in a multitude of ways, but I think it'll work if you want to play along. — 2 hours ago
I'm sure there are ways to make it almost work (I am not the downvoter on your answer, by the way). I just wanted to show that any attempt can be defeated, so this is impossible in the general case. — 2 hours ago

IÃ…ÂŸÃ„Â±k Kaplan 568212 · Answer 4 · 2018-09-15 04:40:37Z

Everyone is writing short codes and meta classes while I barely write a decorator. (damn, lol) But I'm gonna share it anyways.

from functools import wraps


class CLOSED:
 _built_ins = [
 '__add__', '__sub__', '__mul__', '__floordiv__',
 '__div__', '__truediv__', '__mod__', '__divmod__',
 '__pow__', '__lshift__', '__rshift__','__and__',
 '__or__', '__xor__',
 ]

 @staticmethod
 def register_closed(method): # Or you can use type annotations
 method.registered = True # Or you can add the method names as string to closed decorator
 return method # In this version you decorate the methods with this

 @staticmethod
 def closed_method(method, cls):
 @wraps(method)
 def wrapper(*a, **kw):
 return cls(method(*a, **kw))

 return wrapper

 @classmethod
 def closed_class(klass, cls):
 for magic in klass._built_ins:
 _method = getattr(cls, magic, False)
 if _method:
 setattr(cls, magic, klass.closed_method(_method, cls))

 for method in dir(cls):
 c1 = method not in klass._built_ins
 c2 = method not in dir(object)
 c3 = getattr(getattr(cls, method), 'registered', False)
 if all((c1, c2, c3)):
 _method = getattr(cls, method)
 setattr(cls, method, klass.closed_method(_method, cls))
 return cls

Now after you have this long setup, you just decorate the class as you would normally do; I am too sleepy to make it work with inherited classes so for now you have to decorate the class that inherits from a closed class.

@CLOSED.closed_class
class foo(int):
 @CLOSED.register_closed # or if you can simply add this to CLOSED.closed_class
 def bar(self, other): # if you are certain that every method can be casted to its own class
 """Basically just the __add__ method"""
 return self + other


print(type(foo(1) + foo(1))); print(foo(1) + foo(1)) # <class '__main__.foo'> 2
print(type(foo(1).bar(2))); print(foo(1).bar(2)) # <class '__main__.foo'> 3


@CLOSED.closed_class
class baz(foo):
 pass

print(type(baz(1) + baz(3))); print(baz(1) + baz(3)) # <class '__main__.baz'> 4
print(type(baz(1).bar(4))); print(baz(1).bar(4)) # <class '__main__.baz'> 5

Feel free to downvote as I'm still not sure if I understood the question correctly.

score 0 · Answer 5 · 2018-09-15 06:51:52Z

Here is a possible way, also using a metaclass, but just to process the target parameter to the class. The actual magic is done by __getattribute__ and __getattr__, which are generated for the specific target. You will most likely have to register a list of exceptions, which will default to the dunder methods unless you define a exceptions keyword in the class parameter list. For example, __index__, if defined, really has to return an int for your class to work.

 class DunderSet:
 def __contains__(self, x):
 return isinstance(x, string) and x.startswith('__') and x.endswith('__')

 class ClosedMeta(type):
 def __new__(meta, name, bases, dct, **kwargs):
 if 'exceptions' in kwargs:
 exceptions = tuple(map(str, kwargs.pop('exceptions'))
 else:
 exceptions = DunderSet()
 target = kwargs.pop('target', bases[0] if bases else object)
 if kwargs:
 raise ValueError(f'Unknown kwarg(s) ", ".join(kwargs)')
 def __getattr__(self, attr):
 value = target.__getattr__(self, attr)
 if attr in exceptions:
 return value
 if type(value) == target:
 return type(self)(value)
 if callable(value):
 @functools.wraps(value)
 def return_wrapper(*args, **kwargs):
 ret = value(*args, **kwargs)
 return type(self)(ret) if type(ret) == target else ret
 return return_wrapper
 return value

 # ditto for __getattribute__

 dct['__getattr__'] = __getattr__
 dct['__getattribute__'] = __getattribute__
 return super().__new__(meta, name, bases, dct)

This will override any custom __getattr__ or __getattribute__ methods in the class, and it won't close methods that return methods that return the target type. It will also leave non-target return values and attributes completely alone.

And finally, it assumes that you have a wrapper constructor for the target type of the form Proxy(target). You could bypass this assumption by adding or requiring an explicit method (call it __cast__ perhaps).

For a type that extends int, you should not try to override things like __index__, __int__, __trunc__, etc.:

class MyInt(int): pass

class ClosedInt(MyInt, metaclass=ClosedMeta, target=int, exceptions=['__index__', '__int__', '__trunc__']): pass

I've defined the class check on attributes and return values to be exact (as opposed to using isinstance), so you don't need to explicitly list the methods returning a boolean value for example.

The optional target keyword lets you select the target class when it's not the first base.

I'm really curious about the downvote. Is there some coding error I missed? I did write this on mobile after all. Perhaps it's late and I spewed some BS by accident? — 2 hours ago
This isn't going to do anything for magic methods, and there's a bunch of small errors like collections.frozenset or map(string, ...). — 1 hour ago
@user2357112. I've fixed that and a couple of other things. Thanks for the feedback — 21 mins ago
@user2357112. I don't have anything approaching a reasonable testing environment at the moment, so I appreciate your time spent proofreading this. — 1 min ago

score 2 · Answer 6 · 2018-09-15 06:34:52Z

I think using a class decorator with a black list of methods that should not return objects of the same type would be somewhat more Pythonic:

class containerize:
 def __call__(self, obj):
 if isinstance(obj, type):
 return self.decorate_class(obj)
 return self.decorate_callable(obj)

 def decorate_class(self, cls):
 for name in dir(cls):
 attr = getattr(cls, name)
 if callable(attr) and name not in ('__class__', '__init__', '__new__', '__str__', '__repr__', '__getattribute__'):
 setattr(cls, name, self.decorate_callable(attr))
 return cls

 def decorate_callable(self, func):
 def wrapper(obj, *args, **kwargs):
 return obj.__class__(func(obj, *args, **kwargs))
 return wrapper

so that:

class MyInt(int):
 pass

@containerize()
class ClosedIntContainer(MyInt):
 pass

i = ClosedIntContainer(3) + ClosedIntContainer(2)
print(i, type(i))

would output:

5 <class '__main__.ClosedIntContainer'>

score 1 · Answer 7 · 2018-09-15 04:53:37Z

I still feel there might be a more natural way to accomplish this, but I was able to fix the attempt provided in the question.

Here are the main points that needed ot be fixed.

We must check methods of all classes in the mro, not only the bases;

__getattribute__ and __getattr__ must be treated as special cases;

Attributes with __get__ must be treated separately;

We have to write a list of exceptions as methods such as __int__ or __eq__ obviously should return their expected types.

Code

import functools

def get_mro(bases):
 # We omit 'object' as it is the base type
 return type('', bases, ).__mro__[1:-1]

class ClosedMeta(type):
 _register = 

 # Some methods return type must not change
 _exceptions = ('__int__', '__eq__', ...)

 def __new__(cls, name, bases, namespace):
 # A unique id for the class
 uid = max(cls._register) + 1 if cls._register else 0
 mro = get_mro(bases)

 def tail_cast(f):
 """Cast the return value of f"""
 @functools.wraps(f)
 def wrapper(*args, **kwargs):
 out = f(*args, **kwargs)
 if type(out) in mro:
 # Since the class does not exist yet, we will recover it later
 return cls._register[uid](out)
 return out
 return wrapper

 def deep_tail_cast(f):
 """Cast the return value of f or the return value of f(...)"""
 @functools.wraps(f)
 def wrapper(*args, **kwargs):
 out = f(*args, **kwargs)
 if callable(out):
 return tail_cast(out)
 elif type(out) in mro:
 return cls._register[uid](out)
 else:
 return out
 return wrapper

 class PropertyCast:
 """Cast the return value of a property"""
 def __init__(self, prop):
 self.prop = prop

 def __get__(self, instance, owner):
 return cls._register[uid](self.prop.__get__(instance, owner))

 def __set__(self, instance, value):
 return self.prop.__set__(instance, value)

 def __delete__(self, instance):
 return self.prop.__delete__(instance)

 for base in reversed(mro):
 for name, attr in base.__dict__.items():
 if name in ('__getattr__', '__getattribute__'):
 namespace[name] = deep_tail_cast(attr)
 elif callable(attr) and name not in namespace and name not in cls._exceptions:
 namespace[name] = tail_cast(attr)
 elif hasattr(attr, '__get__'):
 namespace[name] = PropertyCast(attr)

 subcls = super().__new__(cls, name, bases, namespace)
 cls._register[uid] = subcls
 return subcls

Example

class MyInt(int):
 def __getattr__(self, _):
 return 1

 @property
 def foo(self):
 return 2

class ClosedInt(MyInt, metaclass=ClosedMeta):
 pass

x = ClosedInt(2)
print(type(x * x), x * x)
print(type(x.foo), x.foo)
print(type(x.bar), x.bar)

Output

<class '__main__.ClosedIntContainer'> 4
<class '__main__.ClosedIntContainer'> 2
<class '__main__.ClosedIntContainer'> 1

This still has some issues. By example we still have the tedious task to go through all dunder methods an mark down exceptions to the implemented rules, but unless there is a list of those somewhere, this seems unavoidable.

score 0 · Answer 8 · 2018-09-15 04:32:54Z

This cannot be done, the data model forbids it. And I can prove it to you:

>>> class MyClass(ClosedInt, metaclass=type):
... def __add__(self, other):
... return 'potato'
... 
>>> MyClass(1) + ClosedInt(2)
'potato'

Addition is handled by the left hand object first, and if the left type handles it (i.e. does not return NotImplemented singleton) then nothing about other is considered in this operation. If the right hand type is a subclass of the left-hand type, you could control the result with the reflected method __radd__ - but of course that is impossible in the general case.

That would be fine since by definition, closure means if x and y are of type T, then x + y is of type T. Here MyClass() is not of type ClosedInt. So the question really applies only if both elements of the operation are of the ClosedInt type — 2 hours ago
Also can not be done, in general case, and the reason is the same - subclasses can override any behavior of parent. — 2 hours ago
@wim. See my attempt. It's not perfect, and of course you can bypass it in a multitude of ways, but I think it'll work if you want to play along. — 2 hours ago
I'm sure there are ways to make it almost work (I am not the downvoter on your answer, by the way). I just wanted to show that any attempt can be defeated, so this is impossible in the general case. — 2 hours ago

IÃ…ÂŸÃ„Â±k Kaplan 568212 · Answer 9 · 2018-09-15 04:40:37Z

Everyone is writing short codes and meta classes while I barely write a decorator. (damn, lol) But I'm gonna share it anyways.

from functools import wraps


class CLOSED:
 _built_ins = [
 '__add__', '__sub__', '__mul__', '__floordiv__',
 '__div__', '__truediv__', '__mod__', '__divmod__',
 '__pow__', '__lshift__', '__rshift__','__and__',
 '__or__', '__xor__',
 ]

 @staticmethod
 def register_closed(method): # Or you can use type annotations
 method.registered = True # Or you can add the method names as string to closed decorator
 return method # In this version you decorate the methods with this

 @staticmethod
 def closed_method(method, cls):
 @wraps(method)
 def wrapper(*a, **kw):
 return cls(method(*a, **kw))

 return wrapper

 @classmethod
 def closed_class(klass, cls):
 for magic in klass._built_ins:
 _method = getattr(cls, magic, False)
 if _method:
 setattr(cls, magic, klass.closed_method(_method, cls))

 for method in dir(cls):
 c1 = method not in klass._built_ins
 c2 = method not in dir(object)
 c3 = getattr(getattr(cls, method), 'registered', False)
 if all((c1, c2, c3)):
 _method = getattr(cls, method)
 setattr(cls, method, klass.closed_method(_method, cls))
 return cls

Now after you have this long setup, you just decorate the class as you would normally do; I am too sleepy to make it work with inherited classes so for now you have to decorate the class that inherits from a closed class.

@CLOSED.closed_class
class foo(int):
 @CLOSED.register_closed # or if you can simply add this to CLOSED.closed_class
 def bar(self, other): # if you are certain that every method can be casted to its own class
 """Basically just the __add__ method"""
 return self + other


print(type(foo(1) + foo(1))); print(foo(1) + foo(1)) # <class '__main__.foo'> 2
print(type(foo(1).bar(2))); print(foo(1).bar(2)) # <class '__main__.foo'> 3


@CLOSED.closed_class
class baz(foo):
 pass

print(type(baz(1) + baz(3))); print(baz(1) + baz(3)) # <class '__main__.baz'> 4
print(type(baz(1).bar(4))); print(baz(1).bar(4)) # <class '__main__.baz'> 5

Feel free to downvote as I'm still not sure if I understood the question correctly.

score 0 · Answer 10 · 2018-09-15 06:51:52Z

Here is a possible way, also using a metaclass, but just to process the target parameter to the class. The actual magic is done by __getattribute__ and __getattr__, which are generated for the specific target. You will most likely have to register a list of exceptions, which will default to the dunder methods unless you define a exceptions keyword in the class parameter list. For example, __index__, if defined, really has to return an int for your class to work.

 class DunderSet:
 def __contains__(self, x):
 return isinstance(x, string) and x.startswith('__') and x.endswith('__')

 class ClosedMeta(type):
 def __new__(meta, name, bases, dct, **kwargs):
 if 'exceptions' in kwargs:
 exceptions = tuple(map(str, kwargs.pop('exceptions'))
 else:
 exceptions = DunderSet()
 target = kwargs.pop('target', bases[0] if bases else object)
 if kwargs:
 raise ValueError(f'Unknown kwarg(s) ", ".join(kwargs)')
 def __getattr__(self, attr):
 value = target.__getattr__(self, attr)
 if attr in exceptions:
 return value
 if type(value) == target:
 return type(self)(value)
 if callable(value):
 @functools.wraps(value)
 def return_wrapper(*args, **kwargs):
 ret = value(*args, **kwargs)
 return type(self)(ret) if type(ret) == target else ret
 return return_wrapper
 return value

 # ditto for __getattribute__

 dct['__getattr__'] = __getattr__
 dct['__getattribute__'] = __getattribute__
 return super().__new__(meta, name, bases, dct)

This will override any custom __getattr__ or __getattribute__ methods in the class, and it won't close methods that return methods that return the target type. It will also leave non-target return values and attributes completely alone.

And finally, it assumes that you have a wrapper constructor for the target type of the form Proxy(target). You could bypass this assumption by adding or requiring an explicit method (call it __cast__ perhaps).

For a type that extends int, you should not try to override things like __index__, __int__, __trunc__, etc.:

class MyInt(int): pass

class ClosedInt(MyInt, metaclass=ClosedMeta, target=int, exceptions=['__index__', '__int__', '__trunc__']): pass

I've defined the class check on attributes and return values to be exact (as opposed to using isinstance), so you don't need to explicitly list the methods returning a boolean value for example.

The optional target keyword lets you select the target class when it's not the first base.

I'm really curious about the downvote. Is there some coding error I missed? I did write this on mobile after all. Perhaps it's late and I spewed some BS by accident? — 2 hours ago
This isn't going to do anything for magic methods, and there's a bunch of small errors like collections.frozenset or map(string, ...). — 1 hour ago
@user2357112. I've fixed that and a couple of other things. Thanks for the feedback — 21 mins ago
@user2357112. I don't have anything approaching a reasonable testing environment at the moment, so I appreciate your time spent proofreading this. — 1 min ago

Search This Blog

Iyfjky

How to create a type that is closed under inherited operations?

5 Answers
5

Code

Example

Output

Your Answer

Post as a guest

5 Answers
5

5 Answers
5

Code

Example

Output

Code

Example

Output

Code

Example

Output

Code

Example

Output

Post as a guest

Comments

Post a Comment

Popular posts from this blog

What does second last employer means? [closed]

Installing NextGIS Connect into QGIS 3?

One-line joke

Category

Random preview

How to create a type that is closed under inherited operations?

5 Answers 5

Code

Example

Output

Your Answer

Sign up or log in

Post as a guest

Post as a guest

5 Answers 5

5 Answers 5

Code

Example

Output

Code

Example

Output

Code

Example

Output

Code

Example

Output

Sign up or log in

Post as a guest

Post as a guest

Sign up or log in

Post as a guest

Sign up or log in

Post as a guest

Sign up or log in

Post as a guest

Comments

Post a Comment

Popular posts from this blog

What does second last employer means? [closed]

Installing NextGIS Connect into QGIS 3?

One-line joke

5 Answers
5

5 Answers
5

5 Answers
5