gitlab升级16.11.3-ee

背景

这是事后一段时间补充记录的博客。

升级目的：修补漏洞CVE-2024-4835 未经认证的威胁攻击者能够利用该漏洞在跨站脚本 (XSS) 攻击中，轻松接管受害者账户。

gitlab版本为14.6.2-ee升级至16.11.3-ee

思路

1. 翻阅文档找升级方法及升级版本路径。
2. 使用备份文件和测试机器模拟升级。但事后发现模拟升级有用，但效果不大
3. 记录测试过程中问题
4. 发邮件通知执行升级时间及影响时长
5. 执行升级

执行

升级路径

根据官网给的升级路径工具指出要跨多个版本 需要先升级至中间版本状态，再升级至目标版本
refer
升级路径工具
https://gitlab-com.gitlab.io/support/toolbox/upgrade-path

执行升级

yum install -y gitlab-ee-14.9.5

升级至14.9.5过程中的问题

acme证书签发问题

gitlab两种https实现，一种是买第三方的https证书手工配置到nginx上，一种是使用gitlab内置的acme一个证书签发管理工具生成https证书实现加密
报错信息
Acme::Client::Error::AccountDoesNotExist: No account exists with the provided key
Error executing action create on resource ‘letsencrypt_certificate[abc.xyz.123]’

Error executing action `create` on resource 'letsencrypt_certificate[abc.xyz.123]'================================================================================Acme::Client::Error::AccountDoesNotExist----------------------------------------acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 41) had an error: Acme::Client::Error::AccountDoesNotExist: No account exists with the provided keyCookbook Trace:---------------/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:58:in `acme_order_certs_for'/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:89:in `block in class_from_file'Resource Declaration:---------------------# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb6: letsencrypt_certificate site do7:   crt node['gitlab']['nginx']['ssl_certificate']8:   key node['gitlab']['nginx']['ssl_certificate_key']9:   notifies :run, "execute[reload nginx]", :immediate10:   notifies :run, 'ruby_block[display_le_message]'11:   only_if { omnibus_helper.service_up?('nginx') }12: endCompiled Resource:------------------# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:6:in `from_file'letsencrypt_certificate("abc.xyz.123") doaction [:create]updated trueupdated_by_last_action truedefault_guard_interpreter :defaultdeclared_type :letsencrypt_certificatecookbook_name "letsencrypt"recipe_name "http_authorization"crt "/etc/gitlab/ssl/abc.xyz.123_cert_chain.pem"key "/etc/gitlab/ssl/abc.xyz.123_key.key"alt_names []cn "abc.xyz.123"only_if { #code block }endSystem Info:------------chef_version=15.17.4platform=centosplatform_version=7.9.2009ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]program_name=/opt/gitlab/embedded/bin/chef-clientexecutable=/opt/gitlab/embedded/bin/chef-clientRecipe: nginx::enable* runit_service[nginx] action restart (up to date)Running handlers:
There was an error running gitlab-ctl reconfigure:letsencrypt_certificate[abc.xyz.123] (letsencrypt::http_authorization line 6) had an error: Acme::Client::Error::AccountDoesNotExist: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 41) had an error: Acme::Client::Error::AccountDoesNotExist: No account exists with the provided keyRunning handlers complete
Chef Infra Client failed. 4 resources updated in 17 secondsj

解决

移除上一次使用的自签名证书

[root@instance-vl1r58a5 acme]# pwd
/etc/acme
[root@instance-vl1r58a5 acme]# mv account_private_key.pem account_private_key.pem.bak20240531
[root@instance-vl1r58a5 acme]# ls
account_private_key.pem  account_private_key.pem.backup20220926  account_private_key.pem.bak20240531
[root@instance-vl1r58a5 acme]# gitlab-ctl reconfigure

refer
https://gbe0.com/posts/linux/server/gitlab-acme-account-does-not-exist/
https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6610
https://forum.gitlab.com/t/gitlab-letsencrypt-issue/63737

继续升级无报错

yum install -y gitlab-ee-14.9.5
yum install -y gitlab-ee-14.10.5
yum install -y gitlab-ee-15.0.5
yum install -y gitlab-ee-15.4.6
yum install -y gitlab-ee-15.11.13

pg数据库报错token设置不符合约束，是gitlab配置的token有效期不能设置为永不过期
报错信息
Error ensuring PostgreSQL is updated. Please check the logs
warning: %posttrans(gitlab-ee-15.11.13-ee.0.el7.x86_64) scriptlet failed, exit status 1
PG::CheckViolation: ERROR: check constraint “check_70f294ef54” of relation “oauth_access_tokens” is violated by some row

main: -- quote_column_name("group_ci_variables")rake aborted!
StandardError: An error has occurred, all later migrations canceled:PG::CheckViolation: ERROR:  check constraint "check_70f294ef54" of relation "oauth_access_tokens" is violated by some row
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:119:in `block in validate_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:116:in `validate_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:232:in `validate_not_null_constraint'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230223014251_validate_not_null_constraint_on_oauth_access_tokens_expires_in.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:19:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:117:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'Caused by:
ActiveRecord::StatementInvalid: PG::CheckViolation: ERROR:  check constraint "check_70f294ef54" of relation "oauth_access_tokens" is violated by some row
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:119:in `block in validate_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:116:in `validate_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:232:in `validate_not_null_constraint'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230223014251_validate_not_null_constraint_on_oauth_access_tokens_expires_in.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:19:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:117:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'Caused by:
PG::CheckViolation: ERROR:  check constraint "check_70f294ef54" of relation "oauth_access_tokens" is violated by some row
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:119:in `block in validate_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:116:in `validate_check_constraint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:232:in `validate_not_null_constraint'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230223014251_validate_not_null_constraint_on_oauth_access_tokens_expires_in.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:19:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:117:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)main:    -> 0.0000s
main: -- quote("early_adopter")
main:    -> 0.0000s
main: -- quote(30000)
main:    -> 0.0000s
main: -- execute("INSERT INTO plan_limits (plan_id, \"group_ci_variables\")\nSELECT id, '30000' FROM plans WHERE name = 'early_adopter' LIMIT 1\nON CONFLICT (plan_id) DO UPDATE SET \"group_ci_variables\" = EXCLUDED.\"group_ci_variables\";\n")
main:    -> 0.0007s
main: -- quote_column_name("group_ci_variables")
main:    -> 0.0000s
main: -- quote("opensource")
main:    -> 0.0000s
main: -- quote(30000)
main:    -> 0.0000s
main: -- execute("INSERT INTO plan_limits (plan_id, \"group_ci_variables\")\nSELECT id, '30000' FROM plans WHERE name = 'opensource' LIMIT 1\nON CONFLICT (plan_id) DO UPDATE SET \"group_ci_variables\" = EXCLUDED.\"group_ci_variables\";\n")
main:    -> 0.0009s
main: == 20230221162222 RaiseCiVariableLimitsOnGitlabCom: migrated (0.0220s) ========main: == 20230221214519 RemoveIncorrectlyOnboardedNamespacesFromOnboardingProgress: migrating
main: == 20230221214519 RemoveIncorrectlyOnboardedNamespacesFromOnboardingProgress: migrated (0.0499s)main: == 20230222035805 PrepareAsyncIndexRemovalOfTokenForCiBuilds: migrating =======
main: -- index_exists?(:ci_builds, :token_encrypted, {:name=>:index_ci_builds_on_token_encrypted})
main:    -> 0.0183s
main: -- quote_column_name(:index_ci_builds_on_token_encrypted)
main:    -> 0.0000s
main: == 20230222035805 PrepareAsyncIndexRemovalOfTokenForCiBuilds: migrated (0.0280s)main: == 20230222055510 RemoveConcurrentIndexOnTokenEncryptedForCiBuilds: migrating =
main: -- transaction_open?()
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0008s
main: -- indexes(:ci_builds)
main:    -> 0.0183s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0003s
main: -- remove_index(:ci_builds, {:algorithm=>:concurrently, :name=>:index_ci_builds_on_token_encrypted})
main:    -> 0.0067s
main: -- execute("RESET statement_timeout")
main:    -> 0.0003s
main: == 20230222055510 RemoveConcurrentIndexOnTokenEncryptedForCiBuilds: migrated (0.0383s)main: == 20230222101420 RemoveFkToCiBuildCiPendingBuildOnBuildId: migrating =========
main: -- transaction_open?()
main:    -> 0.0000s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("LOCK TABLE ci_builds, ci_pending_builds IN ACCESS EXCLUSIVE MODE")
main:    -> 0.0004s
main: -- remove_foreign_key(:ci_pending_builds, :ci_builds, {:name=>:fk_rails_725a2644a3})
main:    -> 0.0059s
main: == 20230222101420 RemoveFkToCiBuildCiPendingBuildOnBuildId: migrated (0.0177s)main: == 20230222102421 RemoveFkToCiBuildCiRunningBuildOnBuildId: migrating =========
main: -- transaction_open?()
main:    -> 0.0000s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("LOCK TABLE ci_builds, ci_running_builds IN ACCESS EXCLUSIVE MODE")
main:    -> 0.0004s
main: -- remove_foreign_key(:ci_running_builds, :ci_builds, {:name=>:fk_rails_da45cfa165})
main:    -> 0.0058s
main: == 20230222102421 RemoveFkToCiBuildCiRunningBuildOnBuildId: migrated (0.0179s)main: == 20230222153048 AddRegistrySizeEstimatedToNamespaceRootStorageStatistics: migrating
main: -- transaction_open?()
main:    -> 0.0000s
main: -- add_column(:namespace_root_storage_statistics, :registry_size_estimated, :boolean, {:default=>false, :null=>false})
main:    -> 0.0012s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0008s
main: -- index_exists?(:namespace_root_storage_statistics, :registry_size_estimated, {:name=>"index_ns_root_stor_stats_on_registry_size_estimated", :algorithm=>:concurrently})
main:    -> 0.0024s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0002s
main: -- add_index(:namespace_root_storage_statistics, :registry_size_estimated, {:name=>"index_ns_root_stor_stats_on_registry_size_estimated", :algorithm=>:concurrently})
main:    -> 0.0044s
main: -- execute("RESET statement_timeout")
main:    -> 0.0003s
main: == 20230222153048 AddRegistrySizeEstimatedToNamespaceRootStorageStatistics: migrated (0.0215s)main: == 20230222161226 AddCustomJiraRegexToJiraTrackerData: migrating ==============
main: -- add_column(:jira_tracker_data, :jira_issue_prefix, :text)
main:    -> 0.0006s
main: -- add_column(:jira_tracker_data, :jira_issue_regex, :text)
main:    -> 0.0005s
main: == 20230222161226 AddCustomJiraRegexToJiraTrackerData: migrated (0.0056s) =====main: == 20230222161954 AddTextLimitToCustomJiraRegexFields: migrating ==============
main: -- transaction_open?()
main:    -> 0.0000s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE jira_tracker_data\nADD CONSTRAINT check_4cc5bbc801\nCHECK ( char_length(jira_issue_prefix) <= 255 )\nNOT VALID;\n")
main:    -> 0.0006s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0008s
main: -- execute("ALTER TABLE jira_tracker_data VALIDATE CONSTRAINT check_4cc5bbc801;")
main:    -> 0.0008s
main: -- execute("RESET statement_timeout")
main:    -> 0.0002s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE jira_tracker_data\nADD CONSTRAINT check_9863a0a5fd\nCHECK ( char_length(jira_issue_regex) <= 255 )\nNOT VALID;\n")
main:    -> 0.0005s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0002s
main: -- execute("ALTER TABLE jira_tracker_data VALIDATE CONSTRAINT check_9863a0a5fd;")
main:    -> 0.0007s
main: -- execute("RESET statement_timeout")
main:    -> 0.0002s
main: == 20230222161954 AddTextLimitToCustomJiraRegexFields: migrated (0.0169s) =====main: == 20230222193845 ChangePublicProjectsMinutesCostFactorDefaultTo1: migrating ==
main: -- transaction_open?()
main:    -> 0.0000s
main: -- change_column_default(:ci_runners, :public_projects_minutes_cost_factor, {:from=>0.0, :to=>1.0})
main:    -> 0.0031s
main: == 20230222193845 ChangePublicProjectsMinutesCostFactorDefaultTo1: migrated (0.0093s)main: == 20230223014251 ValidateNotNullConstraintOnOauthAccessTokensExpiresIn: migrating
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0002s
main: -- execute("ALTER TABLE oauth_access_tokens VALIDATE CONSTRAINT check_70f294ef54;")
main: -- execute("RESET statement_timeout")
main:    -> 0.0002s
STDERR:
---- End output of "bash"  ----
Ran "bash"  returned 1
Running reconfigure: NOT OK
== Fatal error ==
Something went wrong during final reconfiguration, please check the output
== Reverting ==
ok: down: postgresql: 0s, normally up
Symlink correct version of binaries: OK
ok: run: postgresql: (pid 40162) 0s
== Reverted ==
== Reverted to 12.14. Please check output for what went wrong ==
Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: run: alertmanager: (pid 40177) 0s
ok: run: crond: (pid 40188) 1s
ok: run: gitaly: (pid 40196) 0s
ok: run: gitlab-exporter: (pid 40216) 1s
ok: run: gitlab-kas: (pid 40109) 5s
ok: run: grafana: (pid 40220) 0s
ok: run: logrotate: (pid 40231) 0s
ok: run: node-exporter: (pid 40237) 1s
ok: run: postgres-exporter: (pid 40244) 0s
ok: run: prometheus: (pid 40252) 1s
ok: run: redis-exporter: (pid 40263) 0s
ok: run: registry: (pid 40273) 1s
ok: run: sidekiq: (pid 40283) 0s
Toggling services: OK
Checking if a newer PostgreSQL version is available and attempting automatic upgrade to it: NOT OK
Error ensuring PostgreSQL is updated. Please check the logs
warning: %posttrans(gitlab-ee-15.11.13-ee.0.el7.x86_64) scriptlet failed, exit status 1
Non-fatal POSTTRANS scriptlet failure in rpm package gitlab-ee-15.11.13-ee.0.el7.x86_64Verifying  : gitlab-ee-15.11.13-ee.0.el7.x86_64                                                                                                                                  1/2Verifying  : gitlab-ee-15.4.6-ee.0.el7.x86_64                                                                                                                                    2/2Updated:gitlab-ee.x86_64 0:15.11.13-ee.0.el7                                                                                                                                                Complete!

解决

# 停止数据库连接
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiqgitlab-psql -d gitlabhq_production
gitlabhq_production=# SELECT * FROM oauth_access_tokens WHERE expires_in IS NULL;id | resource_owner_id | application_id |                              token                               |                          refresh_token                           | expires_in | revoked_at |         created_at         |  scopes  
----+-------------------+----------------+------------------------------------------------------------------+------------------------------------------------------------------+------------+------------+----------------------------+-----------1 |                 4 |              1 | effcc3b35xxxxx909542e6cdfa34b5528 | 75d6c18ffb3a4ba4xxxxxxxxca27f35 |            |            | 2022-02-17 15:01:40.419753 | read_user
(1 row)gitlabhq_production=# UPDATE oauth_access_tokens SET expires_in = '86400' WHERE expires_in IS NULL;
UPDATE 1
gitlabhq_production=# select * from oauth_access_tokens where expires_in IS NULL;id | resource_owner_id | application_id | token | refresh_token | expires_in | revoked_at | created_at | scopes
----+-------------------+----------------+-------+---------------+------------+------------+------------+--------
(0 rows)

refer
https://docs.gitlab.com/ee/update/versions/gitlab_15_changes.html

pg版本不受支持

解决：升级pg到指定版本13

# 查看当前pg版本
[root@instance-vl1r58a5 ~]# /opt/gitlab/embedded/bin/psql --version
psql (PostgreSQL) 12.14
# 升级至13版本
[root@instance-vl1r58a5 ~]# gitlab-ctl pg-upgrade -V 13
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 13.11
Checking if disk for directory /var/opt/gitlab/postgresql/data has enough free space for PostgreSQL upgrade: OK
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for detailsPlease hit Ctrl-C now if you want to cancel the operation.
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: alertmanager: 0s, normally up
ok: down: crond: 1s, normally up
ok: down: gitaly: 1s, normally up
ok: down: gitlab-exporter: 1s, normally up
ok: down: gitlab-kas: 0s, normally up
ok: down: grafana: 0s, normally up
ok: down: logrotate: 0s, normally up
ok: down: node-exporter: 1s, normally up
ok: down: postgres-exporter: 0s, normally up
ok: down: prometheus: 1s, normally up
ok: down: redis-exporter: 0s, normally up
ok: down: registry: 1s, normally up
ok: down: sidekiq: 0s, normally up
Toggling services: OK
Running stop on postgresql:ok: down: postgresql: 1s, normally up
Running stop on postgresql: OK
Symlink correct version of binaries: OK
Creating temporary data directory: OK
Initializing the new database: OK
Upgrading the data: OK
Move the old data directory out of the way: OK
Rename the new data directory: OK
Saving the old version information: OK
Configuring PostgreSQL
[2024-05-31T23:33:40+08:00] INFO: Started Cinc Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook)
Cinc Client, version 17.10.0
Patents: https://www.chef.io/patents
Infra Phase starting
[2024-05-31T23:33:40+08:00] INFO: *** Cinc Client 17.10.0 ***
[2024-05-31T23:33:40+08:00] INFO: Platform: x86_64-linux
[2024-05-31T23:33:40+08:00] INFO: Cinc-client pid: 3541
[2024-05-31T23:33:42+08:00] INFO: Setting the run_list to ["recipe[gitlab-ee::config]", "recipe[postgresql::enable]"] from CLI options
[2024-05-31T23:33:42+08:00] INFO: Run List is [recipe[gitlab-ee::config], recipe[postgresql::enable]]
[2024-05-31T23:33:42+08:00] INFO: Run List expands to [gitlab-ee::config, postgresql::enable]
[2024-05-31T23:33:42+08:00] INFO: Starting Cinc Client Run for instance-vl1r58a5
[2024-05-31T23:33:42+08:00] INFO: Running start handlers
[2024-05-31T23:33:42+08:00] INFO: Start handlers complete.
Resolving cookbooks for run list: ["gitlab-ee::config", "postgresql::enable"]
[2024-05-31T23:33:43+08:00] INFO: Loading cookbooks [gitlab-ee@0.0.1, postgresql@0.1.0, package@0.1.0, gitlab@0.0.1, consul@0.1.0, patroni@0.1.0, pgbouncer@0.1.0, spamcheck@0.1.0, runit@5.1.7, logrotate@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, acme@4.1.6, crond@0.1.0]
Synchronizing cookbooks:- gitlab-ee (0.0.1)- postgresql (0.1.0)- package (0.1.0)- gitlab (0.0.1)- consul (0.1.0)- patroni (0.1.0)- pgbouncer (0.1.0)- spamcheck (0.1.0)- runit (5.1.7)- logrotate (0.1.0)- redis (0.1.0)- monitoring (0.1.0)- registry (0.1.0)- mattermost (0.1.0)- gitaly (0.1.0)- praefect (0.1.0)- gitlab-kas (0.1.0)- gitlab-pages (0.1.0)- letsencrypt (0.1.0)- nginx (0.1.0)- crond (0.1.0)- acme (4.1.6)
Installing cookbook gem dependencies:
Compiling cookbooks...
Loading Cinc Auditor profile files:Running handlers:
[2024-05-31T23:34:07+08:00] INFO: Running report handlers
Running handlers complete
[2024-05-31T23:34:07+08:00] INFO: Report handlers complete
Infra Phase complete, 8/964 resources updated in 15 seconds
[2024-05-31T23:34:07+08:00] WARN: This release of Cinc Client became end of life (EOL) on May 1st 2023. Please update to a supported release to receive new features, bug fixes, and security updates.
Running reconfigure: OK
Waiting for Database to be running.
Database upgrade is complete, running vacuumdb analyze
Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: run: alertmanager: (pid 4287) 1s
ok: run: crond: (pid 4297) 0s
ok: run: gitaly: (pid 4306) 1s
ok: run: gitlab-exporter: (pid 4325) 0s
ok: run: gitlab-kas: (pid 4328) 0s
ok: run: grafana: (pid 4339) 1s
ok: run: logrotate: (pid 4350) 0s
ok: run: node-exporter: (pid 4359) 1s
ok: run: postgres-exporter: (pid 4366) 0s
ok: run: prometheus: (pid 4378) 1s
ok: run: redis-exporter: (pid 4389) 0s
ok: run: registry: (pid 4397) 0s
ok: run: sidekiq: (pid 4407) 1s
Toggling services: OK
==== Upgrade has completed ====
Please verify everything is working and run the following if so
sudo rm -rf /var/opt/gitlab/postgresql/data.12
sudo rm -f /var/opt/gitlab/postgresql-version.old# 验证
[root@instance-vl1r58a5 ~]# /opt/gitlab/embedded/bin/psql --version
psql (PostgreSQL) 13.11

升级之后要手工合并一下数据，完成数据库迁移

# 合并
[root@instance-vl1r58a5 ~]# gitlab-rake db:migrate
# 在此检查状态ok
[root@instance-vl1r58a5 ~]# gitlab-rake db:migrate:status
# 所有的status状态为up即正常
# 重新加载gitlab配置
gitlab-ctl reconfigure

refer
https://docs.gitlab.com/ee/update/versions/gitlab_15_changes.html
https://docs.gitlab.com/ee/administration/raketasks/maintenance.html#run-incomplete-database-migrations
继续升级，后续升级无报错

yum install -y gitlab-ee-16.3.7
yum install -y gitlab-ee-16.7.7
yum install -y gitlab-ee-16.11.3

其他命令

# 恢复备份命令
gitlab-rake gitlab:backup:restore BACKUP=1621908711
# 查看gitlab版本
cat /opt/gitlab/embedded/service/gitlab-rails/VERSION
gitlab-rake gitlab:env:info

gitlab如果在web端禁用了root用户登录，如下命令启用root用户

gitlabhq_production=# UPDATE users SET state='active' WHERE username='root';
UPDATE 1

修改用户密码
https://docs.gitlab.cn/jh/security/reset_user_password.html
rails控制台

gitlab-rails console
irb(main):006:0> user = User.find_by(username: 'root')
=> #<User id:1 @root>
irb(main):007:0> user.password = 'xxx'
irb(main):008:0> user.password_confirmation = 'xxxx'
irb(main):009:0> user.save
=> true
irb(main):010:0> exit

refer

漏洞信息
https://about.gitlab.com/releases/2024/05/22/patch-release-gitlab-17-0-1-released/
https://gbe0.com/posts/linux/server/gitlab-acme-account-does-not-exist/
https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6610
https://forum.gitlab.com/t/gitlab-letsencrypt-issue/63737
https://docs.gitlab.com/ee/update/versions/gitlab_15_changes.html
https://docs.gitlab.com/ee/update/versions/gitlab_15_changes.html
https://docs.gitlab.com/ee/administration/raketasks/maintenance.html#run-incomplete-database-migrations