Hi, yes I noticed the built in async test support in py3.8, but decided to leave our hack of asynctest due to still having to support 3.7
Testing evolved a lot since we started doing tests here, and the ecosystem was not in place at all. Testing is not going away, but api might change. Especially with pytest doing async so wildly different.

At some point we probably need to rework the test tools. You're welcome to start 😄

Will do

I had a try on tidying up the test cases using unittest but I have to admit, it is not an easy task,
I understand the nature of the changes but I think it can be done differently.

So first and foremost asynctest raises a lot of deprecation errors from python 3.8 and up because of @asyncio.coroutine decorator, so I suggest to switch to the IsolatedAsyncioTestCase implementation copying it for python 3.7 and using it for the later versions of python.

Another thing that I noticed is some methods that are the same as TestCase from asynctest like .init_loop().

So before continuing I want to raise a couple of points with you @grigi and any other contributor that is interested in this matter, so that I do not waste time introducing a change that might not be accepted.

1. Switching to asyncSetUp and asyncTearDown for the async set up and tear down of the test case. To have a more familiar testing coming from the standard python IsolatedAsyncioTestCase.

2. Make find a solution for TestCase for managing the ContexVar in init_connection.
   Here two solutions comes to mind. Adding transactional test case for a single testcase using a decorator (but I do not like this solution). Or reimplement the .run() method of IsolatedAsyncioTestCase like it is done here https://github.com/python/cpython/blob/2b201369b435a4266bda5b895e3b615dbe28ea6e/Lib/unittest/async_case.py#L155 to use a with statement to reuse TransactionTestContext. A solution that in my opinion is much more elegant.

3. This is a question, why all the tests use TestCase and not SimpleTestCase? Most of the tests do not even have more that one test method.

Last but not least, would you be open to move some testing code outside of the contrib module?

I had a try on tidying up the test cases using unittest but I have to admit, it is not an easy task,
I understand the nature of the changes but I think it can be done differently.

Yes, I did mention that this may be the worst code in the entire repo 😭

1. Switching to `asyncSetUp` and `asyncTearDown` for the async set up and tear down of the test case. To have a more familiar testing coming from the standard python `IsolatedAsyncioTestCase`.

Yes, definitely. We would need to have some kind of support for existing customers. Possibly keeping 0.16 alive for a few months to allow migration.

2. Make find a solution for `TestCase` for managing the `ContexVar` in `init_connection`.
   Here two solutions comes to mind. Adding transactional test case for a single testcase using a decorator (but I do not like this solution). Or reimplement the `.run()` method of `IsolatedAsyncioTestCase` like it is done here https://github.com/python/cpython/blob/2b201369b435a4266bda5b895e3b615dbe28ea6e/Lib/unittest/async_case.py#L155 to use a with statement to reuse `TransactionTestContext`. A solution that in my opinion is much more elegant.

The current .run() method seems to destroy and re-create the event loop, which by itself is going to cause a lot of problems for the current test double. It is a thing we must cope with if we ever want to support pytest verb-style tests.
So, until we have a way of migrating our DB connections between loops, we would probably require overriding this. I don't want to re-create the DB between tests as that would significantly slow down testing.

I think the goal is to reduce the amount of boilerplate the user would need for testing, and if simple inheritance can do that for us, then I'd say go for it!

3. This is a question, why all the tests use `TestCase` and not `SimpleTestCase`? Most of the tests do not even have more that one test method.

TestCase ensures that any changes to the DB is rolled back in the most performant way we have (either a transaction or tuncating tables for myisam), whereas SimpleTestCase will leave the detritus in the DB.
Most of the tests actually does a DB interaction, so I would want the DB to be reset as much as possible.
So, I would attribute it to using TestCase by default and using SimpleTestCase when I explicitly need to?

Last but not least, would you be open to move some testing code outside of the contrib module?

Sure. 🤔 That could be part of the solution to allow people to migrate to the new testrunner, we can leave the old, messy one in contrib and raise a DeprecationWarning if you import it with a link to docs.

Where do you want to put it? tortoise.test?

We should look at supporting the minimal work at migrating the test DB between event loops, even if we only use it for some variants of test classes (or pytest support).
It's OK opening and closing connections to persistent databases, but for in-memory SQLite, would prefer to not re-create all the tables, as that is quite slow.

Okay then I will have a try at creating a TestCase in tortoise.test_case, since there might be other testing frameworks that would need some kind of util like pytest or nose (I do not know, never used them, but in aiohttp they have some).

nose/nose2 is just a UnitTest runner, just like green.
The only different one is py.test. It can run standard unittest stuff, but they have their own from-scratch test framework as well.

This is much more complicated than I thought, I am having issues with the implementation of TestCase. It seems like it is related to the nature of the connection of aiosqlite that runs on a thread, so I have multiple loops on multiple threads awaiting the same coroutine. Maybe there is a way to tell it to use the same loop? This is the error

RuntimeError: Task <Task pending name='Task-479' coro=<TransactionWrapper.start() running at /Users/anas/Projects/tortoise-orm/tortoise/backends/sqlite/client.py:171> cb=[_run_until_complete_cb() at /Users/anas/.pyenv/versions/3.8.2/lib/python3.8/asyncio/base_events.py:184] created at /Users/anas/.pyenv/versions/3.8.2/lib/python3.8/asyncio/base_events.py:595> got Future <Future pending> attached to a different loop

Apart from that, this is what I came up with, some notable changes are the adding of db_url,
mainly because I think that the tests of the library should have the same interface of the test cases that the library provides. Another thing that I saw that I did not understand very much is the use of pytest for running unit tests, I mean, that is okay, but why having a session scope fixture?
With that all the other people that want to use the library should adapt to the same pattern?

For the TestCase issue, my main in problem is to find an elegant way to start and rollback a transaction during the test. One possible way would be to add a decorator that wraps all the coroutines. Although complicated it seems.

import inspect
from typing import List
from unittest.async_case import IsolatedAsyncioTestCase

from tortoise import Tortoise, current_transaction_map, generate_config

SimpleTestCase = IsolatedAsyncioTestCase


class IsolatedTestCase(IsolatedAsyncioTestCase):
    """
    An asyncio capable test class that will ensure that an isolated test db
    is available for each test.

    Use this if your test needs perfect isolation.

    Note to use ``{}`` as a string-replacement parameter, for your DB_URL.
    That will create a randomised database name.

    It will create and destroy a new DB instance for every test.
    This is obviously slow, but guarantees a fresh DB.

    If you define a ``tortoise_test_modules`` list, it overrides the DB setup module for the tests.
    """

    tortoise_test_modules: List[str] = []
    db_url: str = ''

    async def asyncSetUp(self) -> None:
        config = generate_config(self.db_url, app_modules={'models': self.tortoise_test_modules},
                                 testing=True, connection_label='models')
        await Tortoise.init(config, _create_db=True)
        await Tortoise.generate_schemas(safe=False)
        self._connections = Tortoise._connections.copy()

    async def asyncTearDown(self) -> None:
        Tortoise._connections = self._connections.copy()
        await Tortoise._drop_databases()


class TruncationTestCase(IsolatedTestCase):
    """
    An asyncio capable test class that will truncate the tables after a test.

    Use this when your tests contain transactions.

    This is slower than ``TestCase`` but faster than ``IsolatedTestCase``.
    Note that usage of this does not guarantee that auto-number-pks will be reset to 1.
    """

    # async def asyncSetUp(self) -> None:
    #     _restore_default()

    async def asyncTearDown(self) -> None:
        Tortoise._connections = self._connections.copy()
        # TODO: This is a naive implementation: Will fail to clear M2M and non-cascade foreign keys
        for app in Tortoise.apps.values():
            for model in app.values():
                await model.all().delete()


class TransactionTestContext:
    __slots__ = ("connection", "connection_name", "token")

    def __init__(self, connection) -> None:
        self.connection = connection
        self.connection_name = connection.connection_name

    async def __aenter__(self):
        current_transaction = current_transaction_map[self.connection_name]
        self.token = current_transaction.set(self.connection)
        if hasattr(self.connection, "_parent"):
            self.connection._connection = await self.connection._parent._pool.acquire()
        await self.connection.start()
        return self.connection

    async def __aexit__(self, exc_type, exc_val, exc_tb) -> None:
        await self.connection.rollback()
        if hasattr(self.connection, "_parent"):
            await self.connection._parent._pool.release(self.connection._connection)
        current_transaction_map[self.connection_name].reset(self.token)


class TestCase(IsolatedTestCase):
    """
    An asyncio capable test class that will ensure that each test will be run at
    separate transaction that will rollback on finish.

    This is a fast test runner. Don't use it if your test uses transactions.
    """

    
    def run(self, result=None):
        self._setupAsyncioLoop()
        try:
            _restore_default()
            self.__db__ = Tortoise.get_connection("models")
            if self.__db__.capabilities.supports_transactions:
                connection = self.__db__._in_transaction().connection
                connection_name = connection.connection_name

                current_transaction = current_transaction_map[connection_name]
                token = current_transaction.set(connection)
                if hasattr(connection, "_parent"):
                    pool = connection._parent._pool
                    connection._connection = self._asyncioTestLoop.run_until_complete(pool.acquire())
                self._asyncioTestLoop.run_until_complete(connection.start())

                result = super().run(result)

                self._asyncioTestLoop.run_until_complete(connection.rollback())
                if hasattr(connection, "_parent"):
                    self._asyncioTestLoop.run_until_complete(connection._parent._pool.release(connection._connection))
                current_transaction_map[connection_name].reset(token)
            else:
                result = super().run(result)

            return result
        finally:
            self._tearDownAsyncioLoop()

    async def asyncSetUp(self) -> None:
        pass

    async def asyncTearDown(self) -> None:
        if self.__db__.capabilities.supports_transactions:
            _restore_default()
        else:
            await super().asyncTearDown()

class TestCase2(IsolatedTestCase):
    """
    An asyncio capable test class that will ensure that each test will be run at
    separate transaction that will rollback on finish.

    This is a fast test runner. Don't use it if your test uses transactions.
    """

    async def _asyncCallWithTransactionContext(self, func, /, *args, **kwargs):
        self.__db__ = Tortoise.get_connection("models")
        if self.__db__.capabilities.supports_transactions:
            connection = self.__db__._in_transaction().connection
            async with TransactionTestContext(connection):
                return await func(*args, **kwargs)
        else:
            return await func(*args, **kwargs)

    async def _callWithTransactionContext(self, func, /, *args, **kwargs):
        return await self._asyncCallWithTransactionContext(func, *args, **kwargs)

    def _callAsync(self, func, /, *args, **kwargs):
        assert self._asyncioTestLoop is not None
        assert inspect.iscoroutinefunction(func)

        ret = self._callWithTransactionContext(func, *args, **kwargs)
        fut = self._asyncioTestLoop.create_future()
        self._asyncioCallsQueue.put_nowait((fut, ret))
        return self._asyncioTestLoop.run_until_complete(fut)

    def _callMaybeAsync(self, func, /, *args, **kwargs):
        assert self._asyncioTestLoop is not None
        if inspect.iscoroutinefunction(func):
            return self._callAsync(func, *args, **kwargs)
        else:
            return func(*args, **kwargs)

Should I make a draft PR?

Sqlite/Mysql/odbc drivers all wrap an existing library in a thread, so this is something we need to support properly. (somehow)
asyncpg, I think, is the only true native asyncio db driver we use right now.

I have not been able to solve that issue either.
As to why we are using pytest to run our tests... It was a concession, but I prefer to run tests locally using green (which is just a unittest runner).
I also don't see utility in the verb-based pytest tests, It forces writing tests in a certain way, which makes doing some kind of tests a lot more verbose and or sometimes nearly impossible to write.
I don't see why unittest is getting the flak it does.

Looking at your proposed code, I have to say it looks a lot less hacky than what we currently have, but as long as the .run() creates a new event loop for every call, I'm uncertain if we will ever get the thread-wrapped drivers to behave?

Your testcase2 seems to be wrapping the test in what is basically the same as tortoise.transactions.atomic?

I think you can create a proposed PR, and we then try and get this fixed? Unfortunately we still have to support Python3.7 for a while, and there we can just use the old testrunner for now? (Unless there is a backport?)

I think the tests of the whole project should be ruined with plain unittest to serve as an example for people approaching the library, for this reason I would like to remove the whole hack of global variables in the initializer.

I think we can get them to behave, my main issue is that the setUp is called after I create the transaction context, a decorator with little modification would solve in my opinion, by decorating all the tests in a TestCase.

Would prefer to not override the logic of IsolatedAsyncioTestCase, I am looking for a small hack that bridges Tortoise to built in python async capable test case.

Your testcase2 seems to be wrapping the test in what is basically the same as tortoise.transactions.atomic?

Yes that something that I will be exploring this weekend.

I think you can create a proposed PR, and we then try and get this fixed? Unfortunately we still have to support Python3.7 for a while, and there we can just use the old testrunner for now? (Unless there is a backport?)

Yes, just a simple copy and paste with a conditional import would do the trick.

I think the tests of the whole project should be ruined with plain unittest to serve as an example for people approaching the library, for this reason I would like to remove the whole hack of global variables in the initializer.

What do you mean by should be ruined?

I think we can get them to behave, my main issue is that the setUp is called after I create the transaction context, a decorator with little modification would solve in my opinion, by decorating all the tests in a TestCase.

Would prefer to not override the logic of IsolatedAsyncioTestCase, I am looking for a small hack that bridges Tortoise to built in python async capable test case.

Would be nice if we can get it to behave whist keeping test performance good. Because I really like being able to run the basic test suite in 4 seconds, instead of 4 minutes.

What do you mean by should be ruined?

Sorry, a non native english speaker mistake, I meant ran. I typed runned which my computer corrected in ruined.

Would be nice if we can get it to behave whist keeping test performance good. Because I really like being able to run the basic test suite in 4 seconds, instead of 4 minutes.

Yes I see your point

Makes so much more sense now 🤣

Don't apologise for non-native english speaker. I think your's is pretty good. My wife keeps on correcting my english 🤷‍♂️

Great work you guys are doing (could only understand 50% of what was being said). Sorry to butt in but are there any plans or progress regarding testing for python 3.8? Just wanted to know if this conversation is still ongoing or if you plan on releasing a dev version (or maybe a hack) for 3.8 users.

Cheers to you guys!

Hello @dropkickdev thanks for showing interest in this,
I experimented with unittest, but I found several issues in making the transactional test.

My idea was to remove the global variable for setup, I did not get back to it, my bad.

There is already some work here but it is still incomplete, I hope to get back to it soon.
If people show interest I will fix it faster and maybe even get some help would be nice

Last but not least, I am going to start some projects with tortoise so I will probably get back to it

Thanks for replying so soon @WisdomPill. Yes, this very much caught my attention and yes those global variables surprised me so much it made me choke on my coffee (true story). I haven't made a fork yet but merely a clone so I could analyze it offline.

I really appreciate the effort you've put into this especially since tests are important for upgrades.

@WisdomPill would you recommend I use your fork instead of the main line for python 3.8? Tortoise works fine it's just the testing that's got me rattled.

I had some issues with transactional tests because of context vars and I did not complete, I think there is still work to be done.

Do you remember what error message it was while working on transactions? Maybe I can take a look.

You know what, I'll go ahead and fork your fork. 🤓 Although I sincerely hope they do something about this. Things can get really scary without any testing.

Hey @grigi @WisdomPill if any of you are in the Philippines hit me up. We can get a beer.

No, I do not remember, I think a lot has to be rewritten, we can work on the same PR instead of forking?

P.S. Thanks for the invitation but I am far away from Philippines, very far, on the other side of the planet in europe

Hi,
I found this issue due to the enormous amount of deprecation errors when running tests with tortoise-orm.
Additionally, there is this issue where the author of asynctest confirms the library is abandoned: Martiusweb/asynctest#158

Shouldn't asynctest be added to tortoise-orm's main dependencies? I see that it is included in the dev dependencies, but if someone wants to write tests for their app that uses tortoise, it needs to be installed.

Thanks for the great work!

Hi guys.
I used the suggested code for isolation tests.

class IsolatedTestCase(IsolatedAsyncioTestCase):
    """
    An asyncio capable test class that will ensure that an isolated test db
    is available for each test.

    Use this if your test needs perfect isolation.

    Note to use ``{}`` as a string-replacement parameter, for your DB_URL.
    That will create a randomised database name.

    It will create and destroy a new DB instance for every test.
    This is obviously slow, but guarantees a fresh DB.

    If you define a ``tortoise_test_modules`` list, it overrides the DB setup module for the tests.
    """

    tortoise_test_modules: List[str] = []
    db_url: str = ''

    async def asyncSetUp(self) -> None:
        config = generate_config(self.db_url, app_modules={'models': self.tortoise_test_modules},
                                 testing=True, connection_label='models')
        await Tortoise.init(config, _create_db=True)
        await Tortoise.generate_schemas(safe=False)
        self._connections = Tortoise._connections.copy()

    async def asyncTearDown(self) -> None:
        Tortoise._connections = self._connections.copy()
        await Tortoise._drop_databases()

Everything is good while I use strawberry(Graphql) schema in the test classes. But when I use TestClient for testing the FastAPI endpoints, I face this error.

    return await self.model.get_or_none(id=user_id)
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/tortoise/queryset.py:966: in _execute
    instance_list = await self._db.executor_class(
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/tortoise/backends/base/executor.py:129: in execute_select
    _, raw_results = await self.db.execute_query(query.get_sql())
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/tortoise/backends/asyncpg/client.py:36: in translate_exceptions_
    return await func(self, *args)
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/tortoise/backends/asyncpg/client.py:185: in execute_query
    rows = await connection.fetch(*params)
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/tortoise/backends/base/client.py:308: in __aexit__
    await self.pool.release(self.connection)
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/asyncpg/pool.py:833: in release
    return await asyncio.shield(ch.release(timeout))
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/asyncpg/pool.py:218: in release
    raise ex
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/asyncpg/pool.py:208: in release
    await self._con.reset(timeout=budget)
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/asyncpg/connection.py:1347: in reset
    await self.execute(reset_query, timeout=timeout)
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/asyncpg/connection.py:315: in execute
    return await self._protocol.query(query, timeout)
asyncpg/protocol/protocol.pyx:323: in query
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   asyncpg.exceptions._base.InterfaceError: cannot perform operation: another operation is in progress

asyncpg/protocol/protocol.pyx:707: InterfaceError

Do you have any idea?

Hi guys. I used the suggested code for isolation tests.

class IsolatedTestCase(IsolatedAsyncioTestCase):
    """
    An asyncio capable test class that will ensure that an isolated test db
    is available for each test.

    Use this if your test needs perfect isolation.

    Note to use ``{}`` as a string-replacement parameter, for your DB_URL.
    That will create a randomised database name.

    It will create and destroy a new DB instance for every test.
    This is obviously slow, but guarantees a fresh DB.

    If you define a ``tortoise_test_modules`` list, it overrides the DB setup module for the tests.
    """

    tortoise_test_modules: List[str] = []
    db_url: str = ''

    async def asyncSetUp(self) -> None:
        config = generate_config(self.db_url, app_modules={'models': self.tortoise_test_modules},
                                 testing=True, connection_label='models')
        await Tortoise.init(config, _create_db=True)
        await Tortoise.generate_schemas(safe=False)
        self._connections = Tortoise._connections.copy()

    async def asyncTearDown(self) -> None:
        Tortoise._connections = self._connections.copy()
        await Tortoise._drop_databases()

Everything is good while I use strawberry(Graphql) schema in the test classes. But when I use TestClient for testing the FastAPI endpoints, I face this error.

    return await self.model.get_or_none(id=user_id)
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/tortoise/queryset.py:966: in _execute
    instance_list = await self._db.executor_class(
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/tortoise/backends/base/executor.py:129: in execute_select
    _, raw_results = await self.db.execute_query(query.get_sql())
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/tortoise/backends/asyncpg/client.py:36: in translate_exceptions_
    return await func(self, *args)
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/tortoise/backends/asyncpg/client.py:185: in execute_query
    rows = await connection.fetch(*params)
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/tortoise/backends/base/client.py:308: in __aexit__
    await self.pool.release(self.connection)
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/asyncpg/pool.py:833: in release
    return await asyncio.shield(ch.release(timeout))
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/asyncpg/pool.py:218: in release
    raise ex
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/asyncpg/pool.py:208: in release
    await self._con.reset(timeout=budget)
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/asyncpg/connection.py:1347: in reset
    await self.execute(reset_query, timeout=timeout)
/home/ahmad/.cache/pypoetry/virtualenvs/authentication-servie-Md6RkHAQ-py3.8/lib/python3.8/site-packages/asyncpg/connection.py:315: in execute
    return await self._protocol.query(query, timeout)
asyncpg/protocol/protocol.pyx:323: in query
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   asyncpg.exceptions._base.InterfaceError: cannot perform operation: another operation is in progress

asyncpg/protocol/protocol.pyx:707: InterfaceError

Do you have any idea?

I used xhttp instead of TestClient and it worked.

But for this we should give up 3.7 in CI?

see #977