服务器断电导致gitlab一直重启，postgresql服务timeout

机房停电后导致gitlab服务器直接宕机了，来电后重启发现gitlab一直起不来，反复的重启

问题排查

• 进入容器内部

docker exec -it star_gitlab2 bash

• 尝试手动重启gitlab所有服务，可以看到postgresql服务没有起来，报错：timeout: run: postgresql: (pid 349) 37s

gitlab-ctl restart

• 查看postgresql日志

gitlab-ctl tail postgresql

日志如下：

root@01bda8746dc4:/# gitlab-ctl tail postgresql
==> /var/log/gitlab/postgresql/current <==
2023-05-29_06:28:27.58283 LOG:  starting PostgreSQL 13.6 on x86_64-pc-linux-gnu,compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0,64-bit
2023-05-29_06:28:27.70593 LOG:  listening on Unix socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"
2023-05-29_06:28:28.09156 LOG:  database system was interrupted; last known up at 2023-05-27 15:06:54 GMT
2023-05-29_06:28:37.45721 FATAL:  the database system is starting up
2023-05-29_06:28:37.46723 FATAL:  the database system is starting up
2023-05-29_06:28:37.48803 FATAL:  the database system is starting up
2023-05-29_06:28:39.39401 FATAL:  the database system is starting up

==> /var/log/gitlab/postgresql/state <==

==> /var/log/gitlab/postgresql/current <==
2023-05-29_06:28:52.30766 FATAL:  the database system is starting up
2023-05-29_06:28:52.30916 FATAL:  the database system is starting up
2023-05-29_06:28:52.31089 FATAL:  the database system is starting up

解决办法

• 切换到postgresql的用户：

su - gitlab-psql

• 执行日志reset：

/opt/gitlab/embedded/bin/pg_resetwal -f /var/opt/gitlab/postgresql/data/

执行后如有可能报错：

pg_resetwal: error: lock file "postmaster.pid" exists
pg_resetwal: Is a server running?  If not,delete the lock file and try again.

按照提示删除"postmaster.pid",重新执行reset即可

cd /var/opt/gitlab/postgresql/data/
rm postmaster.pid

• 重启gitlab所有服务

gitlab-ctl restart

可以看到所有服务均已正常启动，等待几分钟即可访问gitlab了

• 查看状态

root@d63f7cf13aff:/# gitlab-ctl status
run: alertmanager: (pid 880) 1648s; run: log: (pid 338) 1741s
run: gitaly: (pid 892) 1647s; run: log: (pid 324) 1741s
run: gitlab-exporter: (pid 918) 1646s; run: log: (pid 329) 1741s
run: gitlab-kas: (pid 921) 1645s; run: log: (pid 326) 1741s
run: gitlab-workhorse: (pid 934) 1645s; run: log: (pid 322) 1741s
run: logrotate: (pid 948) 1644s; run: log: (pid 333) 1741s
run: nginx: (pid 954) 1644s; run: log: (pid 336) 1741s
run: postgres-exporter: (pid 966) 1644s; run: log: (pid 321) 1741s
run: postgresql: (pid 982) 1642s; run: log: (pid 339) 1741s
run: prometheus: (pid 1003) 1641s; run: log: (pid 332) 1741s
run: puma: (pid 1020) 1641s; run: log: (pid 323) 1741s
run: redis: (pid 1033) 1640s; run: log: (pid 328) 1741s
run: redis-exporter: (pid 1035) 1640s; run: log: (pid 320) 1741s
run: sidekiq: (pid 1129) 1638s; run: log: (pid 319) 1741s
run: sshd: (pid 1135) 1638s; run: log: (pid 30) 1757s

原文地址：https://blog.csdn.net/qq_43347021/article/details/130928562