Can't access web GUI after reboot

@Thad wrote:

Brief description of the problem

After a reboot, I can no longer access the web GUI. SSH works.

Detailed step by step instructions to reproduce the problem

When I attempt to access Rockstor via HTTPS, I get an error.

Web-UI screenshot

“Unable to connect” error message.

Error Traceback provided on the Web-UI

The system boots to a command line and I’m able to SSH in, but it won’t connect via web.

I’m booting from a USB drive (/dev/sda), which appears to be working. There seems to be some corruption on sdb, where the rock-on data is stored; a btrfs check /dev/sdb produces errors like the following:

parent transid verify failed on 12369920 wanted 2531442 found 2531450
Ignoring transid failure
parent transid verify failed on 12369920 wanted 2531442 found 2531450
Ignoring transid failure

Csum exists for 57777917952-57777922048 but there is no extent record
There are no extents for csum range 57780281344-57780285440
Csum exists for 57780281344-57780285440 but there is no extent record
There are no extents for csum range 57780293632-57780301824

Those two sets of errors appear many times with different numerical values.

And here’s the output of tail -n 100 /opt/rockstor/var/log/rockstor.log:

[28/Mar/2020 20:06:47] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_202001241150']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:47] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_201912270750']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:47] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_202001250250']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:47] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_202001300950']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:48] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_201912280750']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:48] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_202002050450']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:48] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_202001280050']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:48] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-daily_202001270130']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:48] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_201912190850']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:48] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_202001281950']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:11:22] ERROR [storageadmin.middleware:32] Exception occurred while processing a request. Path: /api/commands/bootstrap method: POST
[28/Mar/2020 20:11:23] ERROR [storageadmin.middleware:33] FATAL:  the database system is in recovery mode
Traceback (most recent call last):
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 449, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/transaction.py", line 271, in __exit__
    connection.set_autocommit(True)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 294, in set_autocommit
    self.ensure_connection()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 119, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/opt/rockstor/eggs/psycopg2-2.6-py2.7-linux-x86_64.egg/psycopg2/__init__.py", line 164, in connect
    conn = _connect(dsn, connection_factory=connection_factory, async=async)
OperationalError: FATAL:  the database system is in recovery mode

[28/Mar/2020 20:45:48] ERROR [storageadmin.middleware:32] Exception occurred while processing a request. Path: /login_submit method: POST
[28/Mar/2020 20:45:50] ERROR [storageadmin.middleware:33] could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Traceback (most recent call last):
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/rockstor/src/rockstor/storageadmin/views/home.py", line 37, in login_submit
    user = authenticate(username=username, password=password)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/__init__.py", line 74, in authenticate
    user = backend.authenticate(**credentials)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/backends.py", line 17, in authenticate
    user = UserModel._default_manager.get_by_natural_key(username)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/models.py", line 162, in get_by_natural_key
    return self.get(**{self.model.USERNAME_FIELD: username})
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 328, in get
    num = len(clone)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 144, in __len__
    self._fetch_all()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/sql/compiler.py", line 838, in execute_sql
    cursor = self.connection.cursor()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 164, in cursor
    cursor = self.make_cursor(self._cursor())
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 135, in _cursor
    self.ensure_connection()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 119, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/opt/rockstor/eggs/psycopg2-2.6-py2.7-linux-x86_64.egg/psycopg2/__init__.py", line 164, in connect
    conn = _connect(dsn, connection_factory=connection_factory, async=async)
OperationalError: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

[28/Mar/2020 20:50:12] ERROR [scripts.scheduled_tasks.snapshot:133] Failed to create snapshot at shares/12/snapshots/ncmain-hourly_202003282050
[28/Mar/2020 20:50:13] ERROR [scripts.scheduled_tasks.snapshot:134] Exception while setting access_token for url(http://127.0.0.1:8000): HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: /o/token/ (Caused by <class 'socket.error'>: [Errno 111] Connection refused). content: None
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/scripts/scheduled_tasks/snapshot.py", line 128, in main
    save_error=False)
  File "/opt/rockstor/src/rockstor/cli/api_wrapper.py", line 75, in api_call
    self.set_token()
  File "/opt/rockstor/src/rockstor/cli/api_wrapper.py", line 69, in set_token
    raise Exception(msg)
Exception: Exception while setting access_token for url(http://127.0.0.1:8000): HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: /o/token/ (Caused by <class 'socket.error'>: [Errno 111] Connection refused). content: None

Output of systemctl status -l rockstor-pre rockstor rockstor-bootstrap:

● rockstor-pre.service - Tasks required prior to starting Rockstor
   Loaded: loaded (/etc/systemd/system/rockstor-pre.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sat 2020-03-28 22:51:30 MST; 19min ago
  Process: 3051 ExecStart=/opt/rockstor/bin/initrock (code=exited, status=1/FAILURE)
 Main PID: 3051 (code=exited, status=1/FAILURE)

Mar 28 22:51:30 rockstor initrock[3051]: sys.exit(scripts.initrock.main())
Mar 28 22:51:30 rockstor initrock[3051]: File "/opt/rockstor/src/rockstor/scripts/initrock.py", line 392, in main
Mar 28 22:51:30 rockstor initrock[3051]: fake_initial_migration_cmd + ['--database=default', 'contenttypes'])
Mar 28 22:51:30 rockstor initrock[3051]: File "/opt/rockstor/src/rockstor/system/osi.py", line 121, in run_command
Mar 28 22:51:30 rockstor initrock[3051]: raise CommandException(cmd, out, err, rc)
Mar 28 22:51:30 rockstor initrock[3051]: system.exceptions.CommandException: Error running a command. cmd = /opt/rockstor/bin/django migrate --noinput --fake-initial --database=default contenttypes. rc = 1. stdout = ['']. stderr = ['Traceback (most recent call last):', '  File "/opt/rockstor/bin/django", line 44, in <module>', "    sys.exit(djangorecipe.manage.main('rockstor.settings'))", '  File "/opt/rockstor/eggs/djangorecipe-1.9-py2.7.egg/djangorecipe/manage.py", line 9, in main', '    management.execute_from_command_line(sys.argv)', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/__init__.py", line 354, in execute_from_command_line', '    utility.execute()', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/__init__.py", line 346, in execute', '    self.fetch_command(subcommand).run_from_argv(self.argv)', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/base.py", line 394, in run_from_argv', '    self.execute(*args, **cmd_options)', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/base.py", line 445, in execute', '    output = self.handle(*args, **options)', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/commands/migrate.py", line 93, in handle', '    executor = MigrationExecutor(connection, self.migration_progress_callback)', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/executor.py", line 19, in __init__', '    self.loader = MigrationLoader(self.connection)', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/loader.py", line 47, in __init__', '    self.build_graph()', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/loader.py", line 191, in build_graph', '    self.applied_migrations = recorder.applied_migrations()', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/recorder.py", line 59, in applied_migrations', '    self.ensure_schema()', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/recorder.py", line 49, in ensure_schema', '    if self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()):', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 164, in cursor', '    cursor = self.make_cursor(self._cursor())', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 135, in _cursor', '    self.ensure_connection()', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection', '    self.connect()', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__', '    six.reraise(dj_exc_type, dj_exc_value, traceback)', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection', '    self.connect()', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 119, in connect', '    self.connection = self.get_new_connection(conn_params)', '  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection', '    connection = Database.connect(**conn_params)', '  File "/opt/rockstor/eggs/psycopg2-2.6-py2.7-linux-x86_64.egg/psycopg2/__init__.py", line 164, in connect', '    conn = _connect(dsn, connection_factory=connection_factory, async=async)', 'django.db.utils.OperationalError: FATAL:  could not write init file', '', '']
Mar 28 22:51:30 rockstor systemd[1]: rockstor-pre.service: main process exited, code=exited, status=1/FAILURE
Mar 28 22:51:30 rockstor systemd[1]: Failed to start Tasks required prior to starting Rockstor.
Mar 28 22:51:30 rockstor systemd[1]: Unit rockstor-pre.service entered failed state.
Mar 28 22:51:30 rockstor systemd[1]: rockstor-pre.service failed.

● rockstor.service - RockStor startup script
   Loaded: loaded (/etc/systemd/system/rockstor.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

Mar 28 22:51:30 rockstor systemd[1]: Dependency failed for RockStor startup script.
Mar 28 22:51:30 rockstor systemd[1]: Job rockstor.service/start failed with result 'dependency'.

● rockstor-bootstrap.service - Rockstor bootstrapping tasks
   Loaded: loaded (/etc/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: disabled)
   Active: inactive (dead)

Mar 28 22:51:30 rockstor systemd[1]: Dependency failed for Rockstor bootstrapping tasks.
Mar 28 22:51:30 rockstor systemd[1]: Job rockstor-bootstrap.service/start failed with result 'dependency'.

Posts: 1

Participants: 1

Read full topic