Skip to content

introduce user groups#1621

Open
t-lenz wants to merge 10 commits intocms-dev:mainfrom
ioi-germany:groups
Open

introduce user groups#1621
t-lenz wants to merge 10 commits intocms-dev:mainfrom
ioi-germany:groups

Conversation

@t-lenz
Copy link

@t-lenz t-lenz commented Jan 18, 2026

We've been using CMS for the German IOI selection for many years now, and we often have offsite contestants who cannot compete at the same time as the onsite contestants, e.g. because they are ill at the time of the contest or because they live in a different timezone.

If one wants to allot different time slots for the users of a contest in vanilla CMS, this would require setting delay (and possibly extratime) for users individually. This can get pretty inconvenient, in particular if there are multiple contestants affected. Moreover, this does not allow having one group of contestants (in the above example, the onsite contestants) compete in a fixed timeslot and others in a timeslot of their choice (USACO style).

This pull request changes the DB format to introduce user groups. Participations are now assigned a group, and start, stop, per_user_time, etc. are no longer properties of a contest, but instead of a user group. In the above example usecase, one could then simply have one group for the onsite contestants and one for offsite contestants, and one could e.g. have the first group compete at a fixed time, with the offsite group being able to (more or less) freely choose a timeslot. As a proof of concept, we have also adjusted the Italian yaml loader so that it is able to assign users to groups; old yaml configs are still valid and result in all users being assigned to the same group.

This is based on code that has been in use in the German fork of CMS for over 10 years now, but has been cleaned up and updated for the latest version of CMS. Most of the original code was written by @fagu and @magula; the present version also contains contributions by @chuyang-wang-dev.

much of this is based on code originally written by @fagu and @magula
Copy link
Member

@prandla prandla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi, sorry for taking this long to get around to this!

overall, we (or well, at least Luca and i) agree that this would be a good feature to have. i left some comments, some of them minor, some of them requiring more work. i could do all of these fixes myself too, though then i think someone else will need to re-review it :)

in addition to the inline comments, we will need to fix the test suite failures (and possibly add new tests, but currently we don't have a good way to write tests for AWS UI, so i think it's fine to skip that for now). also, we need a DumpUpdater and an sql migration. also, you should run Ruff on modified lines, and some places could use more type hints.

and please do let me know if you disagree with some of the proposed changes, i'm always up for some healthy debate :)

cms/db/user.py Outdated
__table_args__ = (UniqueConstraint("contest_id", "user_id"),)

# Group this user belongs to
group_id = Column(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you added this group_id column, but there is still the participation.contest column. this seems like a normalization violation to me. it seems we should delete participation.contest and use participation.group.contest instead (or possibly add an @property such that participation.contest is a shorthand for participation.group.contest).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually somewhat tricky. Right now, there is a UniquenessConstraint on the pair (user_id, contest_id) in the Participation table to ensure that each user has at most one participation per contest. As far as I understand, SQL would not allow a UniquenessConstraint on the pair (user_id, group.contest_id) since this involves more than one table. As we probably still want to enforce the UniquenessConstraint, my suggestion would be to leave the contest_id and contest columns for Participation (Contest.participations can instead be replaced by a @property as suggested) and to use a ForeignKeyConstraint to ensure that (group_id, contest_id) for Participation matches with (id, contest_id) for Group. This way, we would at least ensure that the data is consistent, although one would still have to set contest_id manually.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, i forgot about that unique constraint. in that case yeah, keeping contest_id makes sense.

enforcing it with a foreign key is definitely good enough. i'm not worried about insertion convenience here, just data integrity.

Should fix all failures except DumpImporterTest and schema_diff_test
(which both likely need more work).
@codecov
Copy link

codecov bot commented Feb 14, 2026

❌ 6 Tests Failed:

Tests completed Failed Passed Skipped
695 6 689 8
View the top 3 failed test(s) by shortest run time
cmstestsuite/unit_tests/cmscontrib/DumpImporterTest.py::TestDumpImporter::test_import_skip_generated
Stack Traces | 0.017s run time
self = <sqlalchemy.engine.base.Connection object at 0x7feeab4de6d0>
dialect = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
constructor = <bound method DefaultExecutionContext._init_compiled of <class 'sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2'>>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 8, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
args = (<sqlalchemy.dialects.postgresql.psycopg2.PGCompiler_psycopg2 object at 0x7feeac92d610>, [{'contest_id': 8, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}])
conn = <sqlalchemy.pool.base._ConnectionFairy object at 0x7feeab757910>
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feeab4ded90>

    def _execute_context(
        self, dialect, constructor, statement, parameters, *args
    ):
        """Create an :class:`.ExecutionContext` and execute, returning
        a :class:`_engine.ResultProxy`.
    
        """
    
        try:
            try:
                conn = self.__connection
            except AttributeError:
                # escape "except AttributeError" before revalidating
                # to prevent misleading stacktraces in Py3K
                conn = None
            if conn is None:
                conn = self._revalidate_connection()
    
            context = constructor(dialect, self, conn, *args)
        except BaseException as e:
            self._handle_dbapi_exception(
                e, util.text_type(statement), parameters, None, None
            )
    
        if context.compiled:
            context.pre_exec()
    
        cursor, statement, parameters = (
            context.cursor,
            context.statement,
            context.parameters,
        )
    
        if not context.executemany:
            parameters = parameters[0]
    
        if self._has_events or self.engine._has_events:
            for fn in self.dispatch.before_cursor_execute:
                statement, parameters = fn(
                    self,
                    cursor,
                    statement,
                    parameters,
                    context,
                    context.executemany,
                )
    
        if self._echo:
            self.engine.logger.info(statement)
            if not self.engine.hide_parameters:
                self.engine.logger.info(
                    "%r",
                    sql_util._repr_params(
                        parameters, batches=10, ismulti=context.executemany
                    ),
                )
            else:
                self.engine.logger.info(
                    "[SQL parameters hidden due to hide_parameters=True]"
                )
    
        evt_handled = False
        try:
            if context.executemany:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_executemany:
                        if fn(cursor, statement, parameters, context):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_executemany(
                        cursor, statement, parameters, context
                    )
            elif not parameters and context.no_parameters:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute_no_params:
                        if fn(cursor, statement, context):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_execute_no_params(
                        cursor, statement, context
                    )
            else:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute:
                        if fn(cursor, statement, parameters, context):
                            evt_handled = True
                            break
                if not evt_handled:
>                   self.dialect.do_execute(
                        cursor, statement, parameters, context
                    )

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1276: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
cursor = <cursor object at 0x7feeab58ab60; closed: -1>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 8, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feeab4ded90>

    def do_execute(self, cursor, statement, parameters, context=None):
>       cursor.execute(statement, parameters)
E       psycopg2.errors.NotNullViolation: null value in column "group_id" of relation "participations" violates not-null constraint
E       DETAIL:  Failing row contains (4, null, null, 00:00:00, 00:00:00, null, f, f, 8, 4, null, null).

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/default.py:608: NotNullViolation

The above exception was the direct cause of the following exception:

self = <DumpImporterTest.TestDumpImporter testMethod=test_import_skip_generated>

    def test_import_skip_generated(self):
        """Test importing everything but the generated data."""
        self.write_dump(TestDumpImporter.DUMP)
        self.write_files(TestDumpImporter.FILES)
>       self.assertTrue(self.do_import(skip_generated=True))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.../unit_tests/cmscontrib/DumpImporterTest.py:261: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../unit_tests/cmscontrib/DumpImporterTest.py:150: in do_import
    skip_users=skip_users).do_import()
                           ^^^^^^^^^^^
cmscontrib/DumpImporter.py:312: in do_import
    session.flush()
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2540: in flush
    self._flush(objects)
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2681: in _flush
    with util.safe_reraise():
.........................................................../cms/lib/python3.11.../sqlalchemy/util/langhelpers.py:68: in __exit__
    compat.raise_(
.........................................................../cms/lib/python3.11.../sqlalchemy/util/compat.py:182: in raise_
    raise exception
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2642: in _flush
    flush_context.execute()
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/unitofwork.py:422: in execute
    rec.execute(self)
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/unitofwork.py:586: in execute
    persistence.save_obj(
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/persistence.py:239: in save_obj
    _emit_insert_statements(
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/persistence.py:1135: in _emit_insert_statements
    result = cached_connections[connection].execute(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1011: in execute
    return meth(self, multiparams, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.........................................................../cms/lib/python3.11.../sqlalchemy/sql/elements.py:298: in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1124: in _execute_clauseelement
    ret = self._execute_context(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1316: in _execute_context
    self._handle_dbapi_exception(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1510: in _handle_dbapi_exception
    util.raise_(
.........................................................../cms/lib/python3.11.../sqlalchemy/util/compat.py:182: in raise_
    raise exception
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1276: in _execute_context
    self.dialect.do_execute(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
cursor = <cursor object at 0x7feeab58ab60; closed: -1>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 8, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feeab4ded90>

    def do_execute(self, cursor, statement, parameters, context=None):
>       cursor.execute(statement, parameters)
E       sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "group_id" of relation "participations" violates not-null constraint
E       DETAIL:  Failing row contains (4, null, null, 00:00:00, 00:00:00, null, f, f, 8, 4, null, null).
E       
E       [SQL: INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, user_id, group_id, team_id) VALUES (CAST(%(ip)s AS CIDR[])::CIDR[], %(starting_time)s, %(delay_time)s, %(extra_time)s, %(password)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id]
E       [parameters: {'ip': None, 'starting_time': None, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'password': None, 'hidden': False, 'unrestricted': False, 'contest_id': 8, 'user_id': 4, 'group_id': None, 'team_id': None}]
E       (Background on this error at: http://sqlalche..../e/13/gkpj)

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/default.py:608: IntegrityError
cmstestsuite/unit_tests/cmscontrib/DumpImporterTest.py::TestDumpImporter::test_import_skip_files
Stack Traces | 0.022s run time
self = <sqlalchemy.engine.base.Connection object at 0x7feeab719150>
dialect = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
constructor = <bound method DefaultExecutionContext._init_compiled of <class 'sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2'>>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 6, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
args = (<sqlalchemy.dialects.postgresql.psycopg2.PGCompiler_psycopg2 object at 0x7feeac92d610>, [{'contest_id': 6, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}])
conn = <sqlalchemy.pool.base._ConnectionFairy object at 0x7feeab7d77d0>
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feeab7d5710>

    def _execute_context(
        self, dialect, constructor, statement, parameters, *args
    ):
        """Create an :class:`.ExecutionContext` and execute, returning
        a :class:`_engine.ResultProxy`.
    
        """
    
        try:
            try:
                conn = self.__connection
            except AttributeError:
                # escape "except AttributeError" before revalidating
                # to prevent misleading stacktraces in Py3K
                conn = None
            if conn is None:
                conn = self._revalidate_connection()
    
            context = constructor(dialect, self, conn, *args)
        except BaseException as e:
            self._handle_dbapi_exception(
                e, util.text_type(statement), parameters, None, None
            )
    
        if context.compiled:
            context.pre_exec()
    
        cursor, statement, parameters = (
            context.cursor,
            context.statement,
            context.parameters,
        )
    
        if not context.executemany:
            parameters = parameters[0]
    
        if self._has_events or self.engine._has_events:
            for fn in self.dispatch.before_cursor_execute:
                statement, parameters = fn(
                    self,
                    cursor,
                    statement,
                    parameters,
                    context,
                    context.executemany,
                )
    
        if self._echo:
            self.engine.logger.info(statement)
            if not self.engine.hide_parameters:
                self.engine.logger.info(
                    "%r",
                    sql_util._repr_params(
                        parameters, batches=10, ismulti=context.executemany
                    ),
                )
            else:
                self.engine.logger.info(
                    "[SQL parameters hidden due to hide_parameters=True]"
                )
    
        evt_handled = False
        try:
            if context.executemany:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_executemany:
                        if fn(cursor, statement, parameters, context):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_executemany(
                        cursor, statement, parameters, context
                    )
            elif not parameters and context.no_parameters:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute_no_params:
                        if fn(cursor, statement, context):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_execute_no_params(
                        cursor, statement, context
                    )
            else:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute:
                        if fn(cursor, statement, parameters, context):
                            evt_handled = True
                            break
                if not evt_handled:
>                   self.dialect.do_execute(
                        cursor, statement, parameters, context
                    )

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1276: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
cursor = <cursor object at 0x7feeab7084f0; closed: -1>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 6, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feeab7d5710>

    def do_execute(self, cursor, statement, parameters, context=None):
>       cursor.execute(statement, parameters)
E       psycopg2.errors.NotNullViolation: null value in column "group_id" of relation "participations" violates not-null constraint
E       DETAIL:  Failing row contains (3, null, null, 00:00:00, 00:00:00, null, f, f, 6, 3, null, null).

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/default.py:608: NotNullViolation

The above exception was the direct cause of the following exception:

self = <DumpImporterTest.TestDumpImporter testMethod=test_import_skip_files>

    def test_import_skip_files(self):
        """Test importing the json but not the files."""
        self.write_dump(TestDumpImporter.DUMP)
        self.write_files(TestDumpImporter.FILES)
>       self.assertTrue(self.do_import(load_files=False))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.../unit_tests/cmscontrib/DumpImporterTest.py:277: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../unit_tests/cmscontrib/DumpImporterTest.py:150: in do_import
    skip_users=skip_users).do_import()
                           ^^^^^^^^^^^
cmscontrib/DumpImporter.py:312: in do_import
    session.flush()
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2540: in flush
    self._flush(objects)
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2681: in _flush
    with util.safe_reraise():
.........................................................../cms/lib/python3.11.../sqlalchemy/util/langhelpers.py:68: in __exit__
    compat.raise_(
.........................................................../cms/lib/python3.11.../sqlalchemy/util/compat.py:182: in raise_
    raise exception
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2642: in _flush
    flush_context.execute()
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/unitofwork.py:422: in execute
    rec.execute(self)
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/unitofwork.py:586: in execute
    persistence.save_obj(
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/persistence.py:239: in save_obj
    _emit_insert_statements(
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/persistence.py:1135: in _emit_insert_statements
    result = cached_connections[connection].execute(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1011: in execute
    return meth(self, multiparams, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.........................................................../cms/lib/python3.11.../sqlalchemy/sql/elements.py:298: in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1124: in _execute_clauseelement
    ret = self._execute_context(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1316: in _execute_context
    self._handle_dbapi_exception(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1510: in _handle_dbapi_exception
    util.raise_(
.........................................................../cms/lib/python3.11.../sqlalchemy/util/compat.py:182: in raise_
    raise exception
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1276: in _execute_context
    self.dialect.do_execute(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
cursor = <cursor object at 0x7feeab7084f0; closed: -1>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 6, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feeab7d5710>

    def do_execute(self, cursor, statement, parameters, context=None):
>       cursor.execute(statement, parameters)
E       sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "group_id" of relation "participations" violates not-null constraint
E       DETAIL:  Failing row contains (3, null, null, 00:00:00, 00:00:00, null, f, f, 6, 3, null, null).
E       
E       [SQL: INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, user_id, group_id, team_id) VALUES (CAST(%(ip)s AS CIDR[])::CIDR[], %(starting_time)s, %(delay_time)s, %(extra_time)s, %(password)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id]
E       [parameters: {'ip': None, 'starting_time': None, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'password': None, 'hidden': False, 'unrestricted': False, 'contest_id': 6, 'user_id': 3, 'group_id': None, 'team_id': None}]
E       (Background on this error at: http://sqlalche..../e/13/gkpj)

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/default.py:608: IntegrityError
cmstestsuite/unit_tests/cmscontrib/DumpImporterTest.py::TestDumpImporter::test_import_old
Stack Traces | 0.052s run time
self = <sqlalchemy.engine.base.Connection object at 0x7feeab67a950>
dialect = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
constructor = <bound method DefaultExecutionContext._init_compiled of <class 'sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2'>>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 4, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
args = (<sqlalchemy.dialects.postgresql.psycopg2.PGCompiler_psycopg2 object at 0x7feeac92d610>, [{'contest_id': 4, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}])
conn = <sqlalchemy.pool.base._ConnectionFairy object at 0x7feeab67bc50>
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feeab678510>

    def _execute_context(
        self, dialect, constructor, statement, parameters, *args
    ):
        """Create an :class:`.ExecutionContext` and execute, returning
        a :class:`_engine.ResultProxy`.
    
        """
    
        try:
            try:
                conn = self.__connection
            except AttributeError:
                # escape "except AttributeError" before revalidating
                # to prevent misleading stacktraces in Py3K
                conn = None
            if conn is None:
                conn = self._revalidate_connection()
    
            context = constructor(dialect, self, conn, *args)
        except BaseException as e:
            self._handle_dbapi_exception(
                e, util.text_type(statement), parameters, None, None
            )
    
        if context.compiled:
            context.pre_exec()
    
        cursor, statement, parameters = (
            context.cursor,
            context.statement,
            context.parameters,
        )
    
        if not context.executemany:
            parameters = parameters[0]
    
        if self._has_events or self.engine._has_events:
            for fn in self.dispatch.before_cursor_execute:
                statement, parameters = fn(
                    self,
                    cursor,
                    statement,
                    parameters,
                    context,
                    context.executemany,
                )
    
        if self._echo:
            self.engine.logger.info(statement)
            if not self.engine.hide_parameters:
                self.engine.logger.info(
                    "%r",
                    sql_util._repr_params(
                        parameters, batches=10, ismulti=context.executemany
                    ),
                )
            else:
                self.engine.logger.info(
                    "[SQL parameters hidden due to hide_parameters=True]"
                )
    
        evt_handled = False
        try:
            if context.executemany:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_executemany:
                        if fn(cursor, statement, parameters, context):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_executemany(
                        cursor, statement, parameters, context
                    )
            elif not parameters and context.no_parameters:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute_no_params:
                        if fn(cursor, statement, context):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_execute_no_params(
                        cursor, statement, context
                    )
            else:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute:
                        if fn(cursor, statement, parameters, context):
                            evt_handled = True
                            break
                if not evt_handled:
>                   self.dialect.do_execute(
                        cursor, statement, parameters, context
                    )

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1276: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
cursor = <cursor object at 0x7feeab58ae30; closed: -1>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 4, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feeab678510>

    def do_execute(self, cursor, statement, parameters, context=None):
>       cursor.execute(statement, parameters)
E       psycopg2.errors.NotNullViolation: null value in column "group_id" of relation "participations" violates not-null constraint
E       DETAIL:  Failing row contains (2, null, null, 00:00:00, 00:00:00, null, f, f, 4, 2, null, null).

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/default.py:608: NotNullViolation

The above exception was the direct cause of the following exception:

self = <DumpImporterTest.TestDumpImporter testMethod=test_import_old>

    def test_import_old(self):
        """Test importing an old dump.
    
        This does not pretend to be exhaustive, just makes sure the happy
        path of the updaters run successfully.
    
        """
        self.write_dump({
            "contest_key": {
                "_class": "Contest",
                "name": "contestname",
                "description": "contest description",
                "start": 1_234_567_890.000,
                "stop": 1_324_567_890.000,
                "token_initial": 2,
                "token_gen_number": 1,
                "token_gen_time": 10,
                "token_total": 100,
                "token_max": 100,
                "tasks": ["task_key"],
            },
            "task_key": {
                "_class": "Task",
                "name": "taskname",
                "title": "task title",
                "num": 0,
                "primary_statements": "[\"en\", \"ja\"]",
                "token_initial": None,
                "token_gen_number": 0,
                "token_gen_time": 0,
                "token_total": None,
                "token_max": None,
                "task_type": "Batch",
                "task_type_parameters": "[]",
                "score_type": "Sum",
                "score_type_parameters": "[]",
                "time_limit": 0.0,
                "memory_limit": None,
                "contest": "contest_key",
                "attachments": {},
                "managers": {},
                "testcases": {},
                "submissions": ["sub1_key", "sub2_key"],
                "user_tests": [],
            },
            "user_key": {
                "_class": "User",
                "username": "username",
                "first_name": "First Name",
                "last_name": "Last Name",
                "password": "pwd",
                "email": "",
                "ip": "0.0.0.0",
                "preferred_languages": "[\"en\", \"it_IT\"]",
                "contest": "contest_key",
                "submissions": ["sub1_key", "sub2_key"],
            },
            "sub1_key": {
                "_class": "Submission",
                "timestamp": 1_234_567_890.123,
                "language": "c",
                "user": "user_key",
                "task": "task_key",
                "compilation_text": "OK [1.234 - 20]",
                "files": {},
                "executables": {"exe": "exe_key"},
                "evaluations": [],
            },
            "sub2_key": {
                "_class": "Submission",
                "timestamp": 1_234_567_900.123,
                "language": "c",
                "user": "user_key",
                "task": "task_key",
                "compilation_text": "Killed with signal 11 [0.123 - 10]\n",
                "files": {},
                "executables": {},
                "evaluations": [],
            },
            "exe_key": {
                "_class": "Executable",
                "submission": "sub1_key",
                "filename": "exe",
                "digest": TestDumpImporter.GENERATED_FILE_DIGEST,
            },
            "_version": 1,
            "_objects": ["contest_key", "user_key"],
        })
        self.write_files(TestDumpImporter.FILES)
>       self.assertTrue(self.do_import(skip_generated=True))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.../unit_tests/cmscontrib/DumpImporterTest.py:394: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../unit_tests/cmscontrib/DumpImporterTest.py:150: in do_import
    skip_users=skip_users).do_import()
                           ^^^^^^^^^^^
cmscontrib/DumpImporter.py:312: in do_import
    session.flush()
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2540: in flush
    self._flush(objects)
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2681: in _flush
    with util.safe_reraise():
.........................................................../cms/lib/python3.11.../sqlalchemy/util/langhelpers.py:68: in __exit__
    compat.raise_(
.........................................................../cms/lib/python3.11.../sqlalchemy/util/compat.py:182: in raise_
    raise exception
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2642: in _flush
    flush_context.execute()
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/unitofwork.py:422: in execute
    rec.execute(self)
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/unitofwork.py:586: in execute
    persistence.save_obj(
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/persistence.py:239: in save_obj
    _emit_insert_statements(
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/persistence.py:1135: in _emit_insert_statements
    result = cached_connections[connection].execute(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1011: in execute
    return meth(self, multiparams, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.........................................................../cms/lib/python3.11.../sqlalchemy/sql/elements.py:298: in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1124: in _execute_clauseelement
    ret = self._execute_context(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1316: in _execute_context
    self._handle_dbapi_exception(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1510: in _handle_dbapi_exception
    util.raise_(
.........................................................../cms/lib/python3.11.../sqlalchemy/util/compat.py:182: in raise_
    raise exception
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1276: in _execute_context
    self.dialect.do_execute(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
cursor = <cursor object at 0x7feeab58ae30; closed: -1>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 4, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feeab678510>

    def do_execute(self, cursor, statement, parameters, context=None):
>       cursor.execute(statement, parameters)
E       sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "group_id" of relation "participations" violates not-null constraint
E       DETAIL:  Failing row contains (2, null, null, 00:00:00, 00:00:00, null, f, f, 4, 2, null, null).
E       
E       [SQL: INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, user_id, group_id, team_id) VALUES (CAST(%(ip)s AS CIDR[])::CIDR[], %(starting_time)s, %(delay_time)s, %(extra_time)s, %(password)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id]
E       [parameters: {'ip': None, 'starting_time': None, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'password': None, 'hidden': False, 'unrestricted': False, 'contest_id': 4, 'user_id': 2, 'group_id': None, 'team_id': None}]
E       (Background on this error at: http://sqlalche..../e/13/gkpj)

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/default.py:608: IntegrityError
cmstestsuite/unit_tests/cmscontrib/DumpImporterTest.py::TestDumpImporter::test_import
Stack Traces | 0.158s run time
self = <sqlalchemy.engine.base.Connection object at 0x7feead9b3090>
dialect = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
constructor = <bound method DefaultExecutionContext._init_compiled of <class 'sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2'>>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 2, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
args = (<sqlalchemy.dialects.postgresql.psycopg2.PGCompiler_psycopg2 object at 0x7feeac92d610>, [{'contest_id': 2, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}])
conn = <sqlalchemy.pool.base._ConnectionFairy object at 0x7feeab7c1150>
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feead9cd950>

    def _execute_context(
        self, dialect, constructor, statement, parameters, *args
    ):
        """Create an :class:`.ExecutionContext` and execute, returning
        a :class:`_engine.ResultProxy`.
    
        """
    
        try:
            try:
                conn = self.__connection
            except AttributeError:
                # escape "except AttributeError" before revalidating
                # to prevent misleading stacktraces in Py3K
                conn = None
            if conn is None:
                conn = self._revalidate_connection()
    
            context = constructor(dialect, self, conn, *args)
        except BaseException as e:
            self._handle_dbapi_exception(
                e, util.text_type(statement), parameters, None, None
            )
    
        if context.compiled:
            context.pre_exec()
    
        cursor, statement, parameters = (
            context.cursor,
            context.statement,
            context.parameters,
        )
    
        if not context.executemany:
            parameters = parameters[0]
    
        if self._has_events or self.engine._has_events:
            for fn in self.dispatch.before_cursor_execute:
                statement, parameters = fn(
                    self,
                    cursor,
                    statement,
                    parameters,
                    context,
                    context.executemany,
                )
    
        if self._echo:
            self.engine.logger.info(statement)
            if not self.engine.hide_parameters:
                self.engine.logger.info(
                    "%r",
                    sql_util._repr_params(
                        parameters, batches=10, ismulti=context.executemany
                    ),
                )
            else:
                self.engine.logger.info(
                    "[SQL parameters hidden due to hide_parameters=True]"
                )
    
        evt_handled = False
        try:
            if context.executemany:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_executemany:
                        if fn(cursor, statement, parameters, context):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_executemany(
                        cursor, statement, parameters, context
                    )
            elif not parameters and context.no_parameters:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute_no_params:
                        if fn(cursor, statement, context):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_execute_no_params(
                        cursor, statement, context
                    )
            else:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute:
                        if fn(cursor, statement, parameters, context):
                            evt_handled = True
                            break
                if not evt_handled:
>                   self.dialect.do_execute(
                        cursor, statement, parameters, context
                    )

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1276: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
cursor = <cursor object at 0x7feeab7084f0; closed: -1>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 2, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feead9cd950>

    def do_execute(self, cursor, statement, parameters, context=None):
>       cursor.execute(statement, parameters)
E       psycopg2.errors.NotNullViolation: null value in column "group_id" of relation "participations" violates not-null constraint
E       DETAIL:  Failing row contains (1, null, null, 00:00:00, 00:00:00, null, f, f, 2, 1, null, null).

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/default.py:608: NotNullViolation

The above exception was the direct cause of the following exception:

self = <DumpImporterTest.TestDumpImporter testMethod=test_import>

    def test_import(self):
        """Test importing everything, while keeping the existing contest."""
        self.write_dump(TestDumpImporter.DUMP)
        self.write_files(TestDumpImporter.FILES)
>       self.assertTrue(self.do_import())
                        ^^^^^^^^^^^^^^^^

.../unit_tests/cmscontrib/DumpImporterTest.py:224: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../unit_tests/cmscontrib/DumpImporterTest.py:150: in do_import
    skip_users=skip_users).do_import()
                           ^^^^^^^^^^^
cmscontrib/DumpImporter.py:312: in do_import
    session.flush()
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2540: in flush
    self._flush(objects)
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2681: in _flush
    with util.safe_reraise():
.........................................................../cms/lib/python3.11.../sqlalchemy/util/langhelpers.py:68: in __exit__
    compat.raise_(
.........................................................../cms/lib/python3.11.../sqlalchemy/util/compat.py:182: in raise_
    raise exception
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2642: in _flush
    flush_context.execute()
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/unitofwork.py:422: in execute
    rec.execute(self)
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/unitofwork.py:586: in execute
    persistence.save_obj(
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/persistence.py:239: in save_obj
    _emit_insert_statements(
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/persistence.py:1135: in _emit_insert_statements
    result = cached_connections[connection].execute(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1011: in execute
    return meth(self, multiparams, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.........................................................../cms/lib/python3.11.../sqlalchemy/sql/elements.py:298: in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1124: in _execute_clauseelement
    ret = self._execute_context(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1316: in _execute_context
    self._handle_dbapi_exception(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1510: in _handle_dbapi_exception
    util.raise_(
.........................................................../cms/lib/python3.11.../sqlalchemy/util/compat.py:182: in raise_
    raise exception
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1276: in _execute_context
    self.dialect.do_execute(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
cursor = <cursor object at 0x7feeab7084f0; closed: -1>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 2, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feead9cd950>

    def do_execute(self, cursor, statement, parameters, context=None):
>       cursor.execute(statement, parameters)
E       sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "group_id" of relation "participations" violates not-null constraint
E       DETAIL:  Failing row contains (1, null, null, 00:00:00, 00:00:00, null, f, f, 2, 1, null, null).
E       
E       [SQL: INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, user_id, group_id, team_id) VALUES (CAST(%(ip)s AS CIDR[])::CIDR[], %(starting_time)s, %(delay_time)s, %(extra_time)s, %(password)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id]
E       [parameters: {'ip': None, 'starting_time': None, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'password': None, 'hidden': False, 'unrestricted': False, 'contest_id': 2, 'user_id': 1, 'group_id': None, 'team_id': None}]
E       (Background on this error at: http://sqlalche..../e/13/gkpj)

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/default.py:608: IntegrityError
cmstestsuite/unit_tests/cmscontrib/DumpImporterTest.py::TestDumpImporter::test_import_with_drop
Stack Traces | 0.21s run time
self = <sqlalchemy.engine.base.Connection object at 0x7feeab680790>
dialect = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
constructor = <bound method DefaultExecutionContext._init_compiled of <class 'sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2'>>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 1, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
args = (<sqlalchemy.dialects.postgresql.psycopg2.PGCompiler_psycopg2 object at 0x7feeac92d610>, [{'contest_id': 1, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}])
conn = <sqlalchemy.pool.base._ConnectionFairy object at 0x7feeab681010>
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feeab7ce0d0>

    def _execute_context(
        self, dialect, constructor, statement, parameters, *args
    ):
        """Create an :class:`.ExecutionContext` and execute, returning
        a :class:`_engine.ResultProxy`.
    
        """
    
        try:
            try:
                conn = self.__connection
            except AttributeError:
                # escape "except AttributeError" before revalidating
                # to prevent misleading stacktraces in Py3K
                conn = None
            if conn is None:
                conn = self._revalidate_connection()
    
            context = constructor(dialect, self, conn, *args)
        except BaseException as e:
            self._handle_dbapi_exception(
                e, util.text_type(statement), parameters, None, None
            )
    
        if context.compiled:
            context.pre_exec()
    
        cursor, statement, parameters = (
            context.cursor,
            context.statement,
            context.parameters,
        )
    
        if not context.executemany:
            parameters = parameters[0]
    
        if self._has_events or self.engine._has_events:
            for fn in self.dispatch.before_cursor_execute:
                statement, parameters = fn(
                    self,
                    cursor,
                    statement,
                    parameters,
                    context,
                    context.executemany,
                )
    
        if self._echo:
            self.engine.logger.info(statement)
            if not self.engine.hide_parameters:
                self.engine.logger.info(
                    "%r",
                    sql_util._repr_params(
                        parameters, batches=10, ismulti=context.executemany
                    ),
                )
            else:
                self.engine.logger.info(
                    "[SQL parameters hidden due to hide_parameters=True]"
                )
    
        evt_handled = False
        try:
            if context.executemany:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_executemany:
                        if fn(cursor, statement, parameters, context):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_executemany(
                        cursor, statement, parameters, context
                    )
            elif not parameters and context.no_parameters:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute_no_params:
                        if fn(cursor, statement, context):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_execute_no_params(
                        cursor, statement, context
                    )
            else:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute:
                        if fn(cursor, statement, parameters, context):
                            evt_handled = True
                            break
                if not evt_handled:
>                   self.dialect.do_execute(
                        cursor, statement, parameters, context
                    )

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1276: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
cursor = <cursor object at 0x7feeab58b100; closed: -1>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 1, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feeab7ce0d0>

    def do_execute(self, cursor, statement, parameters, context=None):
>       cursor.execute(statement, parameters)
E       psycopg2.errors.NotNullViolation: null value in column "group_id" of relation "participations" violates not-null constraint
E       DETAIL:  Failing row contains (1, null, null, 00:00:00, 00:00:00, null, f, f, 1, 1, null, null).

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/default.py:608: NotNullViolation

The above exception was the direct cause of the following exception:

self = <DumpImporterTest.TestDumpImporter testMethod=test_import_with_drop>

    def test_import_with_drop(self):
        """Test importing everything, but dropping existing data."""
        self.write_dump(TestDumpImporter.DUMP)
        self.write_files(TestDumpImporter.FILES)
    
        # Need to close the session and reopen it, otherwise the drop hangs.
        self.session.close()
>       self.assertTrue(self.do_import(drop=True))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^

.../unit_tests/cmscontrib/DumpImporterTest.py:244: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../unit_tests/cmscontrib/DumpImporterTest.py:150: in do_import
    skip_users=skip_users).do_import()
                           ^^^^^^^^^^^
cmscontrib/DumpImporter.py:312: in do_import
    session.flush()
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2540: in flush
    self._flush(objects)
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2681: in _flush
    with util.safe_reraise():
.........................................................../cms/lib/python3.11.../sqlalchemy/util/langhelpers.py:68: in __exit__
    compat.raise_(
.........................................................../cms/lib/python3.11.../sqlalchemy/util/compat.py:182: in raise_
    raise exception
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/session.py:2642: in _flush
    flush_context.execute()
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/unitofwork.py:422: in execute
    rec.execute(self)
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/unitofwork.py:586: in execute
    persistence.save_obj(
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/persistence.py:239: in save_obj
    _emit_insert_statements(
.........................................................../cms/lib/python3.11.../sqlalchemy/orm/persistence.py:1135: in _emit_insert_statements
    result = cached_connections[connection].execute(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1011: in execute
    return meth(self, multiparams, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.........................................................../cms/lib/python3.11.../sqlalchemy/sql/elements.py:298: in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1124: in _execute_clauseelement
    ret = self._execute_context(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1316: in _execute_context
    self._handle_dbapi_exception(
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1510: in _handle_dbapi_exception
    util.raise_(
.........................................................../cms/lib/python3.11.../sqlalchemy/util/compat.py:182: in raise_
    raise exception
.........................................................../cms/lib/python3.11.../sqlalchemy/engine/base.py:1276: in _execute_context
    self.dialect.do_execute(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7feeaee22110>
cursor = <cursor object at 0x7feeab58b100; closed: -1>
statement = 'INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, us...d)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id'
parameters = {'contest_id': 1, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'group_id': None, ...}
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x7feeab7ce0d0>

    def do_execute(self, cursor, statement, parameters, context=None):
>       cursor.execute(statement, parameters)
E       sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "group_id" of relation "participations" violates not-null constraint
E       DETAIL:  Failing row contains (1, null, null, 00:00:00, 00:00:00, null, f, f, 1, 1, null, null).
E       
E       [SQL: INSERT INTO participations (ip, starting_time, delay_time, extra_time, password, hidden, unrestricted, contest_id, user_id, group_id, team_id) VALUES (CAST(%(ip)s AS CIDR[])::CIDR[], %(starting_time)s, %(delay_time)s, %(extra_time)s, %(password)s, %(hidden)s, %(unrestricted)s, %(contest_id)s, %(user_id)s, %(group_id)s, %(team_id)s) RETURNING participations.id]
E       [parameters: {'ip': None, 'starting_time': None, 'delay_time': datetime.timedelta(0), 'extra_time': datetime.timedelta(0), 'password': None, 'hidden': False, 'unrestricted': False, 'contest_id': 1, 'user_id': 1, 'group_id': None, 'team_id': None}]
E       (Background on this error at: http://sqlalche..../e/13/gkpj)

.........................................................../cms/lib/python3.11.../sqlalchemy/engine/default.py:608: IntegrityError
cmstestsuite/unit_tests/schema_diff_test.py::TestSchemaDiff::test_schema_diff
Stack Traces | 0.46s run time
self = <cmstestsuite.unit_tests.schema_diff_test.TestSchemaDiff testMethod=test_schema_diff>

    def test_schema_diff(self):
        dirname = os.path.dirname(__file__)
        schema_file = os.path.join(dirname, "schema_v1.5.sql")
        updater_file = os.path.join(dirname, "../...../cmscontrib/updaters/update_from_1.5.sql")
        updated_schema = split_schema(get_updated_schema(schema_file, updater_file))
        fresh_schema = split_schema(get_fresh_schema())
        errors = compare_schemas(updated_schema, fresh_schema)
        self.longMessage = False
>       self.assertTrue(errors == "", errors)
E       AssertionError: Statement differs between updated and fresh schema:
E       CREATE TABLE public.contests (
E       -     CONSTRAINT contests_check CHECK ((start <= stop)),
E       -     CONSTRAINT contests_check1 CHECK ((stop <= analysis_start)),
E       -     CONSTRAINT contests_check2 CHECK ((analysis_start <= analysis_stop)),
E       -     CONSTRAINT contests_check3 CHECK ((token_gen_initial <= token_gen_max)),
E       ?                              -
E       +     CONSTRAINT contests_check CHECK ((token_gen_initial <= token_gen_max)),
E             CONSTRAINT contests_max_submission_number_check CHECK ((max_submission_number > 0)),
E             CONSTRAINT contests_max_user_test_number_check CHECK ((max_user_test_number > 0)),
E             CONSTRAINT contests_min_submission_interval_check CHECK ((min_submission_interval > '00:00:00'::interval)),
E             CONSTRAINT contests_min_submission_interval_grace_period_check CHECK ((min_submission_interval_grace_period > '00:00:00'::interval)),
E             CONSTRAINT contests_min_user_test_interval_check CHECK ((min_user_test_interval > '00:00:00'::interval)),
E             CONSTRAINT contests_per_user_time_check CHECK ((per_user_time >= '00:00:00'::interval)),
E             CONSTRAINT contests_score_precision_check CHECK ((score_precision >= 0)),
E             CONSTRAINT contests_token_gen_initial_check CHECK ((token_gen_initial >= 0)),
E             CONSTRAINT contests_token_gen_interval_check CHECK ((token_gen_interval > '00:00:00'::interval)),
E             CONSTRAINT contests_token_gen_max_check CHECK ((token_gen_max > 0)),
E             CONSTRAINT contests_token_gen_number_check CHECK ((token_gen_number >= 0)),
E             CONSTRAINT contests_token_max_number_check CHECK ((token_max_number > 0)),
E             CONSTRAINT contests_token_min_interval_check CHECK ((token_min_interval >= '00:00:00'::interval)),
E             allow_password_authentication boolean NOT NULL,
E             allow_questions boolean NOT NULL,
E             allow_registration boolean NOT NULL,
E             allow_unofficial_submission_before_analysis_mode boolean NOT NULL,
E             allow_user_tests boolean NOT NULL,
E             allowed_localizations character varying[] NOT NULL,
E       -     analysis_enabled boolean NOT NULL,
E       -     analysis_start timestamp without time zone NOT NULL,
E       -     analysis_stop timestamp without time zone NOT NULL,
E             block_hidden_participations boolean NOT NULL,
E             description character varying NOT NULL,
E             id integer NOT NULL,
E             ip_autologin boolean NOT NULL,
E             ip_restriction boolean NOT NULL,
E             languages character varying[] NOT NULL,
E       +     main_group_id integer,
E             max_submission_number integer,
E             max_user_test_number integer,
E             min_submission_interval interval,
E             min_submission_interval_grace_period interval,
E             min_user_test_interval interval,
E             name public.codename NOT NULL,
E             per_user_time interval,
E             score_precision integer NOT NULL,
E       -     start timestamp without time zone NOT NULL,
E       -     stop timestamp without time zone NOT NULL,
E             submissions_download_allowed boolean NOT NULL,
E             timezone character varying,
E             token_gen_initial integer NOT NULL,
E             token_gen_interval interval NOT NULL,
E             token_gen_max integer,
E             token_gen_number integer NOT NULL,
E             token_max_number integer,
E             token_min_interval interval NOT NULL,
E             token_mode public.token_mode NOT NULL,
E         );
E       Statement differs between updated and fresh schema:
E       CREATE TABLE public.participations (
E             CONSTRAINT participations_delay_time_check CHECK ((delay_time >= '00:00:00'::interval)),
E             CONSTRAINT participations_extra_time_check CHECK ((extra_time >= '00:00:00'::interval)),
E             contest_id integer NOT NULL,
E             delay_time interval NOT NULL,
E             extra_time interval NOT NULL,
E       +     group_id integer NOT NULL,
E             hidden boolean NOT NULL,
E             id integer NOT NULL,
E             ip cidr[],
E             password character varying,
E             starting_time timestamp without time zone,
E             team_id integer,
E             unrestricted boolean NOT NULL,
E             user_id integer NOT NULL,
E         );
E       Fresh schema contains extra statement:
E       CREATE TABLE public.groups (
E           CONSTRAINT groups_check CHECK ((start <= stop)),
E           CONSTRAINT groups_check1 CHECK ((stop <= analysis_start)),
E           CONSTRAINT groups_check2 CHECK ((analysis_start <= analysis_stop)),
E           CONSTRAINT groups_per_user_time_check CHECK ((per_user_time >= '00:00:00'::interval)),
E           analysis_enabled boolean NOT NULL,
E           analysis_start timestamp without time zone NOT NULL,
E           analysis_stop timestamp without time zone NOT NULL,
E           contest_id integer,
E           id integer NOT NULL,
E           name character varying NOT NULL,
E           per_user_time interval,
E           start timestamp without time zone NOT NULL,
E           stop timestamp without time zone NOT NULL,
E       );
E       Fresh schema contains extra statement:
E       ALTER TABLE public.groups OWNER TO postgres;
E       Fresh schema contains extra statement:
E       CREATE SEQUENCE public.groups_id_seq
E           AS integer
E           START WITH 1
E           INCREMENT BY 1
E           NO MINVALUE
E           NO MAXVALUE
E           CACHE 1;
E       Fresh schema contains extra statement:
E       ALTER TABLE public.groups_id_seq OWNER TO postgres;
E       Fresh schema contains extra statement:
E       ALTER SEQUENCE public.groups_id_seq OWNED BY public.groups.id;
E       Fresh schema contains extra statement:
E       ALTER TABLE ONLY public.groups ALTER COLUMN id SET DEFAULT nextval('public.groups_id_seq'::regclass);
E       Fresh schema contains extra statement:
E       ALTER TABLE ONLY public.groups ADD CONSTRAINT groups_contest_id_name_key
E       UNIQUE (contest_id, name);
E       Fresh schema contains extra statement:
E       ALTER TABLE ONLY public.groups ADD CONSTRAINT groups_pkey
E       PRIMARY KEY (id);
E       Fresh schema contains extra statement:
E       CREATE INDEX ix_contests_main_group_id ON public.contests USING btree (main_group_id);
E       Fresh schema contains extra statement:
E       CREATE INDEX ix_groups_contest_id ON public.groups USING btree (contest_id);
E       Fresh schema contains extra statement:
E       CREATE INDEX ix_participations_group_id ON public.participations USING btree (group_id);
E       Fresh schema contains extra statement:
E       ALTER TABLE ONLY public.contests ADD CONSTRAINT fk_contest_main_group_id
E       FOREIGN KEY (main_group_id) REFERENCES public.groups(id) ON UPDATE CASCADE ON DELETE SET NULL;
E       Fresh schema contains extra statement:
E       ALTER TABLE ONLY public.groups ADD CONSTRAINT groups_contest_id_fkey
E       FOREIGN KEY (contest_id) REFERENCES public.contests(id) ON UPDATE CASCADE ON DELETE CASCADE;
E       Fresh schema contains extra statement:
E       ALTER TABLE ONLY public.participations ADD CONSTRAINT participations_group_id_fkey
E       FOREIGN KEY (group_id) REFERENCES public.groups(id) ON UPDATE CASCADE ON DELETE CASCADE;

cmstestsuite/unit_tests/schema_diff_test.py:173: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Copy link
Member

@prandla prandla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noticed a few more things when reading through the code one more time, and poking around in AWS a bit. i think after addressing these comments, and the foreign key constraint mentioned in the previous discussion, the code itself is fine. again we still need a DumpUpdater and a sql updater. i can write those myself if you'd prefer (i assume you haven't needed to deal with those features in your fork...)


from sqlalchemy.dialects.postgresql import ARRAY, CIDR
from sqlalchemy.orm import relationship
from sqlalchemy.orm import backref, relationship
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this import is unnecessary now

Integer,
ForeignKey(Contest.id,
onupdate="CASCADE", ondelete="CASCADE"),
# nullable=False,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have an explicit nullable=True or nullable=False on every column, so why not have it here too?

(i don't know who initially did this, but i like it being explicit because then i don't have to remember what sqlalchemy's default is :) )

foreign_keys=[contest_id],
back_populates="groups")

def phase(self, timestamp: datetime) -> int:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like you copied an older version of phase() from contest.py to here. i'd prefer keeping the comment i added to it.

try:
contest_id: str = self.get_argument("contest_id")
assert contest_id != "null", "Please select a valid contest"
group_id: str = self.get_argument("group_id")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you forgot to update admin/templates/user.html; there's still just a "select contest" dropdown there, which only sets contest_id and thus breaks this endpoint.

attrs["contest"] = self.safe_get_item(Contest, contest_id)

# Create the group.
group = Group(**attrs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer if this handler copied the attributes from the contest's main group, instead of using the hardcoded defaults.


self.get_string(attrs, "name", empty=None)

self.get_datetime(attrs, "start")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'd really like if this code wasn't duplicated with ContestHandler.post(). There's already some differences between them; namely this version doesn't have explicit checks for "please set start/stop time" (i'm not sure if those are strictly necessary or if it's just to have a nicer error than the default IntegrityError, but in any case, there's no reason for these to be different).

something like get_group_settings(handler: BaseHandler, group: Group) which can then be called as get_group_settings(self, group) here and as get_group_settings(self, contest.main_group) in ContestHandler.

team_code: str | None,
hidden: bool,
unrestricted: bool,
groupname: str
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

either this can be None, in which case the hint should be str | None, or it can't, in which case the if groupname is None check is unnecessary. (given all the logic in main() i assume groupname can't be None actually.)


# Groups
main_group_name: str | None = load(conf, None, "main_group")
groups: Group | None = load(conf, None, "groups")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this isn't the right type hint (it's a dict representing group settings, not the actual group object)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants