@phillxnet wrote:
I’ve recently had a report, via support email, of Web-UI and share failure. Bringing this to the forum so that others may chip in. I’m pasting what I think are the most pertinent parts of the submitted logs downloaded from the Web-UI by the reporter. Currently that is the only information I have. I am going to invite the reporter to this thread so we might use the better-than-email forum facilities to have a go at getting them sorted.
All welcome to assist and I’m currently unsure of users technical level; what we have is:
I lost access to my shared folder and GUI, please help me.
I have requested, via email return, the output of the following commands:
btrfs fi showand
btrfs fi usage /mnt2/rockstor_rockstorPasting the pertinent log entries as I see them:
2019-11-28 13:47:53 [10083] [INFO] Listening at: http://127.0.0.1:8000 (10083) 2019-11-28 13:47:53 [10083] [INFO] Using worker: sync 2019-11-28 13:47:53 [10185] [INFO] Booting worker with pid: 10185 2019-11-28 13:47:53 [10184] [INFO] Booting worker with pid: 10184 2019-12-31 18:45:24 [10052] [INFO] Starting gunicorn 0.16.1 2019-12-31 18:45:24 [10052] [INFO] Listening at: http://127.0.0.1:8000 (10052) 2019-12-31 18:45:24 [10052] [INFO] Using worker: sync 2019-12-31 18:45:24 [10156] [INFO] Booting worker with pid: 10156 2019-12-31 18:45:24 [10155] [INFO] Booting worker with pid: 10155 2020-01-20 08:59:58 [10052] [INFO] Handling signal: term 2020-01-20 08:59:58 [10155] [INFO] Worker exiting (pid: 10155) 2020-01-20 08:59:58 [10156] [INFO] Worker exiting (pid: 10156) 2020-01-20 08:59:58 [10052] [INFO] Shutting down: Master Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 875, in emit self.flush() File "/usr/lib64/python2.7/logging/__init__.py", line 835, in flush self.stream.flush() IOError: [Errno 28] No space left on device Logged from file glogging.py, line 184 2020-01-20 09:01:33 [10082] [INFO] Using worker: sync 2020-01-20 09:01:33 [10106] [INFO] Booting worker with pid: 10106 2020-01-20 09:01:33 [10105] [INFO] Booting worker with pid: 10105The above gunicorn.log ends as it does above.
So initially this looks like the system disk is out of space, has gone read only, or is failing; and we also have the following db access happening at the same time:
rockstor.log
DoesNotExist: NetworkConnection matching query does not exist. [20/Jan/2020 08:56:55] ERROR [storageadmin.views.network:157] NetworkConnection matching query does not exist. Traceback (most recent call last): File "/opt/rockstor/src/rockstor/storageadmin/views/network.py", line 154, in update_connection name=dconfig['connection']) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 334, in get self.model._meta.object_name DoesNotExist: NetworkConnection matching query does not exist. [20/Jan/2020 09:01:34] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /o/token/ method: POST [20/Jan/2020 09:01:34] ERROR [storageadmin.middleware:33] FATAL: the database system is in recovery mode Traceback (most recent call last): File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 71, in view return self.dispatch(request, *args, **kwargs) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 34, in _wrapper return bound_func(*args, **kwargs) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/csrf.py", line 58, in wrapped_view return view_func(*args, **kwargs) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 30, in bound_func return func.__get__(self, type(self))(*args2, **kwargs2) File "/opt/rockstor/eggs/django_braces-1.11.0-py2.7.egg/braces/views/_forms.py", line 24, in dispatch return super(CsrfExemptMixin, self).dispatch(*args, **kwargs) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 89, in dispatch return handler(request, *args, **kwargs) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 34, in _wrapper return bound_func(*args, **kwargs) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper return view(request, *args, **kwargs) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 30, in bound_func return func.__get__(self, type(self))(*args2, **kwargs2) File "/opt/rockstor/eggs/django_oauth_toolkit-0.9.0-py2.7.egg/oauth2_provider/views/base.py", line 172, in post url, headers, body, status = self.create_token_response(request) File "/opt/rockstor/eggs/django_oauth_toolkit-0.9.0-py2.7.egg/oauth2_provider/views/mixins.py", line 124, in create_token_response return core.create_token_response(request) File "/opt/rockstor/eggs/django_oauth_toolkit-0.9.0-py2.7.egg/oauth2_provider/oauth2_backends.py", line 126, in create_token_response headers) File "/opt/rockstor/eggs/oauthlib-1.0.1-py2.7.egg/oauthlib/oauth2/rfc6749/endpoints/base.py", line 64, in wrapper return f(endpoint, uri, *args, **kwargs) File "/opt/rockstor/eggs/oauthlib-1.0.1-py2.7.egg/oauthlib/oauth2/rfc6749/endpoints/token.py", line 100, in create_token_response request, self.default_token_type) File "/opt/rockstor/eggs/oauthlib-1.0.1-py2.7.egg/oauthlib/oauth2/rfc6749/grant_types/client_credentials.py", line 80, in create_token_response token = token_handler.create_token(request, refresh_token=False) File "/opt/rockstor/eggs/oauthlib-1.0.1-py2.7.egg/oauthlib/oauth2/rfc6749/tokens.py", line 276, in create_token self.request_validator.save_bearer_token(token, request) File "/opt/rockstor/eggs/django_oauth_toolkit-0.9.0-py2.7.egg/oauth2_provider/oauth2_validators.py", line 310, in save_bearer_token access_token.save() File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 734, in save force_update=force_update, update_fields=update_fields) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 762, in save_base updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/transaction.py", line 271, in __exit__ connection.set_autocommit(True) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 294, in set_autocommit self.ensure_connection() File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection self.connect() File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection self.connect() File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 119, in connect self.connection = self.get_new_connection(conn_params) File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection connection = Database.connect(**conn_params) File "/opt/rockstor/eggs/psycopg2-2.6-py2.7-linux-x86_64.egg/psycopg2/__init__.py", line 164, in connect conn = _connect(dsn, connection_factory=connection_factory, async=async) OperationalError: FATAL: the database system is in recovery modeand a db with no space left may well announce itself to be “… in recovery mode” hence failing Web-UI.
There after we have a repeating:
[20/Jan/2020 09:01:44] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /login_submit method: POST [20/Jan/2020 09:01:44] ERROR [storageadmin.middleware:33] could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"? Traceback (most recent call last): ... [20/Jan/2020 09:01:58] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /login_submit method: POST [20/Jan/2020 09:01:58] ERROR [storageadmin.middleware:33] could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"? Traceback (most recent call last):Further from the supervisord.log we have at the same time:
2019-11-28 13:47:51,766 INFO supervisord started with pid 9700 2019-11-28 13:47:52,769 INFO spawned: 'nginx' with pid 10082 2019-11-28 13:47:52,771 INFO spawned: 'gunicorn' with pid 10083 2019-11-28 13:47:52,773 INFO spawned: 'data-collector' with pid 10084 2019-11-28 13:47:52,775 INFO spawned: 'ztask-daemon' with pid 10085 2019-11-28 13:47:54,804 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs) 2019-11-28 13:47:54,804 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 2 seconds (startsecs) 2019-11-28 13:47:57,807 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) 2019-11-28 13:47:57,808 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) 2019-12-31 18:45:22,556 CRIT Supervisor running as root (no user in config file) 2019-12-31 18:45:22,574 INFO RPC interface 'supervisor' initialized 2019-12-31 18:45:22,576 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2019-12-31 18:45:22,577 INFO supervisord started with pid 9669 2019-12-31 18:45:23,579 INFO spawned: 'nginx' with pid 10051 2019-12-31 18:45:23,582 INFO spawned: 'gunicorn' with pid 10052 2019-12-31 18:45:23,585 INFO spawned: 'data-collector' with pid 10053 2019-12-31 18:45:23,589 INFO spawned: 'ztask-daemon' with pid 10054 2019-12-31 18:45:25,615 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs) 2019-12-31 18:45:25,615 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 2 seconds (startsecs) 2019-12-31 18:45:28,619 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) 2019-12-31 18:45:28,620 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) 2020-01-20 08:59:58,123 WARN received SIGTERM indicating exit request 2020-01-20 08:59:58,123 INFO waiting for data-collector, nginx, gunicorn, ztask-daemon to die 2020-01-20 08:59:58,131 INFO exited: data-collector (terminated by SIGTERM; not expected) 2020-01-20 08:59:58,133 INFO exited: ztask-daemon (terminated by SIGTERM; not expected) 2020-01-20 08:59:58,140 INFO exited: nginx (exit status 0; expected) 2020-01-20 08:59:58,244 INFO exited: gunicorn (exit status 0; expected) 2020-01-20 09:01:31,233 CRIT Supervisor running as root (no user in config file) 2020-01-20 09:01:31,259 INFO RPC interface 'supervisor' initialized 2020-01-20 09:01:31,264 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2020-01-20 09:01:31,265 INFO supervisord started with pid 9294 2020-01-20 09:01:32,269 INFO spawned: 'nginx' with pid 10081 2020-01-20 09:01:32,273 INFO spawned: 'gunicorn' with pid 10082 2020-01-20 09:01:32,275 INFO spawned: 'data-collector' with pid 10083 2020-01-20 09:01:32,278 INFO spawned: 'ztask-daemon' with pid 10084 2020-01-20 09:01:37,364 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs) 2020-01-20 09:01:37,364 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)So again failure of the supporting services to start with the exited entries above.
I may not have much time to address this issue myself so if anyone can chip in that would be great.
Cheers.
Posts: 1
Participants: 1