Clever Geek Handbook

Como restaurar o Sentry após uma falha na atualização

Olá. Eu quero falar sobre como foi a restauração do Sentinela após uma atualização falha.



O que é Sentinela ?



É um sistema completo de rastreamento de bugs de código aberto que oferece suporte a uma ampla gama de linguagens e ambientes do lado do servidor, navegador, desktop e nativos para dispositivos móveis, incluindo PHP, Node. js, Python, Ruby, C #, Java, Go, React, Angular, Vue, JavaScript e muito mais.



Um pouco sobre como tudo quebrou de maneira simples e sem restrições.



Vivíamos na versão 9.0.0 e é hora de atualizar. Tendo sentido a interface web 10.0.0, decidi atualizá-la. Esse foi o maior erro!



A atualização ocorreu normalmente. Primeiro, vá para 9.1.2, depois para 10.0.0 (caso contrário, é impossível). Então descobri que o trabalhador não inicia. A razão para isso é a adição de novos aplicativos e dependências.



As tentativas de construir aplicativos e colocá-los no k8s não tiveram sucesso.



Foi decidido reverter. Mas aqui fiquei desapontado. Após a migração para a versão 10, alguns dos dados são movidos para clickhouse e removidos do Postgres.



Após 3 horas de trabalho e o envolvimento de um especialista em DBA, os dados foram restaurados à condição de trabalho.



Como?



Fizemos uma instância PG próxima a ele, retiramos o esquema da versão 9.0.0 e iniciamos uma recuperação tabular. Claro, houve alguns erros e um arquivo.



A história não termina aí. É hora de atualizar para a versão 9.1.2.



Colete a imagem, execute: 



sentry upgrade


e vemos esta mensagem maravilhosa: 



 ! I'm not trusting myself; either fix this yourself by fiddling
 ! with the south_migrationhistory table, or pass --delete-ghost-migrations
 ! to South to have it delete ALL of these records (this may not be good).
Exception in thread Thread-1 (most likely raised during interpreter shutdown)


Depois de uma longa busca, tentativa e erro, uma solução foi encontrada.



Você precisa restaurar o esquema da versão atual.



Para fazer isso, clonamos o repositório: 



git clone https://github.com/getsentry/onpremise.git


Vá para o branch 9.1.2, já que o branch 9.0.0 acabou. 



cd onpremise && git checkout tags/9.1.2


Em seguida, editamos a imagem no Dockerfile: 



#cat Dockerfile 
ARG SENTRY_IMAGE
FROM ${SENTRY_IMAGE:-sentry:9.0.0}-onbuild
Em seguida, edite o arquivo de composição



docker-compose.yml 
# NOTE: This docker-compose.yml is meant to be just an example of how
# you could accomplish this on your own. It is not intended to work in
# all use-cases and must be adapted to fit your needs. This is merely
# a guideline.

# See docs.getsentry.com/on-premise/server/ for full
# instructions

version: '3.4'

x-defaults: &defaults
  restart: unless-stopped
  build:
    context: .
  depends_on:
    - redis
   # - postgres
    - memcached
    - smtp
  env_file: .env
  environment:
    SENTRY_MEMCACHED_HOST: memcached
    SENTRY_REDIS_HOST: redis
    SENTRY_POSTGRES_HOST: 'sentry.cl.ats'
    SENTRY_DB_USER: 'sentryDB'
    SENTRY_DB_NAME: 'sentry_user'
    SENTRY_DB_PASSWORD: 'sentry_passwd'
    SENTRY_POSTGRES_PORT: 5432
    SENTRY_EMAIL_HOST: smtp
  volumes:
    - sentry-data:/var/lib/sentry/files


services:
  smtp:
    restart: unless-stopped
    image: tianon/exim4

  memcached:
    restart: unless-stopped
    image: memcached:1.5-alpine

  redis:
    restart: unless-stopped
    image: redis:3.2-alpine

  #postgres:
  #  restart: unless-stopped
  #  image: postgres:9.5
  #  environment:
  #    POSTGRES_HOST_AUTH_METHOD: 'trust'
  #  volumes:
  #    - sentry-postgres:/var/lib/postgresql/data

  web:
    <<: *defaults
    ports:
      - '9000:9000'

  cron:
    <<: *defaults
    command: run cron

  worker:
    <<: *defaults
    command: run worker


volumes:
    sentry-data:
      external: true
    #sentry-postgres:
    #  external: true




Poucas explicações. Como tenho uma instância de banco de dados separada, pulo a criação por meio de compose e o volume sentry-postgres também não é necessário.



Após essas manipulações, iniciamos a montagem: 



./install.sh


No processo, pode aparecer um erro, que mencionei acima. Não prestamos atenção e após o término do processo, execute o seguinte comando: 



docker-compose run --rm web sentry django migrate --delete-ghost-migrations


Conclusão aproximada 
Creating network "onpremise_default" with the default driver
Creating onpremise_memcached_1 ... done
Creating onpremise_smtp_1      ... done
Creating onpremise_redis_1     ... done
21:30:26 [INFO] sentry.plugins.github: apps-not-configured
Running migrations for sentry:
- Nothing to migrate.
 - Loading initial data for sentry.
Installed 0 object(s) from 0 fixture(s)
Running migrations for nodestore:
- Nothing to migrate.
 - Loading initial data for nodestore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for search:
- Nothing to migrate.
 - Loading initial data for search.
Installed 0 object(s) from 0 fixture(s)
Running migrations for social_auth:
- Nothing to migrate.
 - Loading initial data for social_auth.
Installed 0 object(s) from 0 fixture(s)
Running migrations for tagstore:
- Nothing to migrate.
 - Loading initial data for tagstore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for jira_ac:
- Nothing to migrate.
 - Loading initial data for jira_ac.
Installed 0 object(s) from 0 fixture(s)
Running migrations for hipchat_ac:
- Nothing to migrate.
 - Loading initial data for hipchat_ac.
Installed 0 object(s) from 0 fixture(s)




Então você pode executar a atualização. 



docker-compose run --rm web sentry upgrade


Stdout 
Starting onpremise_smtp_1      ... done
Starting onpremise_memcached_1 ... done
Starting onpremise_redis_1     ... done
21:30:48 [INFO] sentry.plugins.github: apps-not-configured
Syncing...
Creating tables ...
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

Synced:
 > django.contrib.admin
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.messages
 > django.contrib.sessions
 > django.contrib.sites
 > django.contrib.staticfiles
 > crispy_forms
 > debug_toolbar
 > raven.contrib.django.raven_compat
 > rest_framework
 > sentry.plugins.sentry_interface_types
 > sentry.plugins.sentry_mail
 > sentry.plugins.sentry_urls
 > sentry.plugins.sentry_useragents
 > sentry.plugins.sentry_webhooks
 > sudo
 > south
 > sentry_plugins.slack

Not synced (use migrations):
 - sentry
 - sentry.nodestore
 - sentry.search
 - social_auth
 - sentry.tagstore
 - sentry_plugins.jira_ac
 - sentry_plugins.hipchat_ac
(use ./manage.py migrate to migrate these)
Running migrations for sentry:
- Nothing to migrate.
 - Loading initial data for sentry.
Installed 0 object(s) from 0 fixture(s)
Running migrations for nodestore:
- Nothing to migrate.
 - Loading initial data for nodestore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for search:
- Nothing to migrate.
 - Loading initial data for search.
Installed 0 object(s) from 0 fixture(s)
Running migrations for social_auth:
- Nothing to migrate.
 - Loading initial data for social_auth.
Installed 0 object(s) from 0 fixture(s)
Running migrations for tagstore:
- Nothing to migrate.
 - Loading initial data for tagstore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for jira_ac:
- Nothing to migrate.
 - Loading initial data for jira_ac.
Installed 0 object(s) from 0 fixture(s)
Running migrations for hipchat_ac:
- Nothing to migrate.
 - Loading initial data for hipchat_ac.
Installed 0 object(s) from 0 fixture(s)
Creating missing DSNs
Correcting Group.num_comments counter




Então, o circuito é restaurado. Você pode atualizar para 9.1.2.



Para fazer isso, no Dockerfile, altere a versão para 9.1.2: 



ARG SENTRY_IMAGE
FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
E execute install.sh 



./install.sh


Se tudo for feito corretamente, o processo será concluído sem erros. A saída deve ser algo assim:



Stdout 
# ./install.sh 
Checking minimum requirements...

Creating volumes for persistent storage...
Created sentry-data.

.env already exists, skipped creation.

Building and tagging Docker images...

smtp uses an image, skipping
memcached uses an image, skipping
redis uses an image, skipping
Building web
Step 1/2 : ARG SENTRY_IMAGE
Step 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
# Executing 4 build triggers
 ---> Running in 6c97f9fcaf63
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Removing intermediate container 6c97f9fcaf63
 ---> Running in 9e0f65ee3af6
Removing intermediate container 9e0f65ee3af6
 ---> Running in 09754c44319c
Removing intermediate container 09754c44319c
 ---> a12fa31c2675

Successfully built a12fa31c2675
Successfully tagged onpremise_web:latest
Building cron
Step 1/2 : ARG SENTRY_IMAGE
Step 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
# Executing 4 build triggers
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> a12fa31c2675

Successfully built a12fa31c2675
Successfully tagged onpremise_cron:latest
Building worker
Step 1/2 : ARG SENTRY_IMAGE
Step 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
# Executing 4 build triggers
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> a12fa31c2675

Successfully built a12fa31c2675
Successfully tagged onpremise_worker:latest

Docker images built.

Generating secret key...
Secret key written to .env

Setting up database...
Starting onpremise_smtp_1      ... done
Starting onpremise_redis_1     ... done
Starting onpremise_memcached_1 ... done
21:35:07 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
21:35:10 [INFO] sentry.plugins.github: apps-not-configured
Syncing...
Creating tables ...
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
Migrating...
Running migrations for sentry:
 - Migrating forwards to 0472_auto__add_field_sentryapp_author.
 > sentry:0424_auto__add_field_integration_status
 > sentry:0425_auto__add_index_pullrequest_organization_id_merge_commit_sha
 > sentry:0425_remove_invalid_github_idps
 > sentry:0426_auto__add_sentryappinstallation__add_sentryapp__add_field_user_is_sent
 > sentry:0427_auto__add_eventattachment__add_unique_eventattachment_project_id_event
 > sentry:0428_auto__add_index_eventattachment_project_id_date_added
 > sentry:0429_auto__add_integrationexternalproject__add_unique_integrationexternalpr
 > sentry:0430_auto__add_field_organizationintegration_status
 > sentry:0431_auto__add_field_externalissue_metadata
 > sentry:0432_auto__add_field_relay_is_internal
 > sentry:0432_auto__add_index_userreport_date_added__add_index_eventattachment_date_
 > sentry:0433_auto__add_field_relay_is_internal__add_field_userip_country_code__add_
 > sentry:0434_auto__add_discoversavedqueryproject__add_unique_discoversavedqueryproj
 > sentry:0435_auto__add_field_discoversavedquery_created_by
 > sentry:0436_rename_projectdsymfile_to_projectdebugfile
 > sentry:0437_auto__add_field_sentryapp_status
 > sentry:0438_auto__add_index_sentryapp_status__chg_field_sentryapp_proxy_user__chg_
 > sentry:0439_auto__chg_field_sentryapp_owner
 > sentry:0440_auto__del_unique_projectdebugfile_project_debug_id__add_index_projectd
 > sentry:0441_auto__add_field_projectdebugfile_data
 > sentry:0442_auto__add_projectcficachefile__add_unique_projectcficachefile_project_
 > sentry:0443_auto__add_field_organizationmember_token_expires_at
 > sentry:0443_auto__del_dsymapp__del_unique_dsymapp_project_platform_app_id__del_ver
 > sentry:0444_auto__add_sentryappavatar__add_field_sentryapp_redirect_url__add_field
 > sentry:0445_auto__add_promptsactivity__add_unique_promptsactivity_user_feature_org
 > sentry:0446_auto__add_index_promptsactivity_project_id
 > sentry:0447_auto__del_field_promptsactivity_organization__add_field_promptsactivit
 > sentry:0448_auto__add_field_sentryapp_is_alertable
 > sentry:0449_auto__chg_field_release_owner
 > sentry:0450_auto__del_grouphashtombstone__del_unique_grouphashtombstone_project_ha
 > sentry:0451_auto__del_field_projectbookmark_project_id__add_field_projectbookmark_
 > sentry:0452_auto__add_field_sentryapp_events
 > sentry:0452_auto__del_field_releaseenvironment_organization_id__del_field_releasee
 > sentry:0453_auto__add_index_releasefile_release_name
 > sentry:0454_resolve_duplicate_0452
 > sentry:0455_auto__add_field_groupenvironment_first_seen
 > sentry:0456_auto__add_dashboard__add_unique_dashboard_organization_title__add_widg
 > sentry:0457_auto__add_field_savedsearch_is_global__chg_field_savedsearch_project__
 > sentry:0457_auto__add_monitorcheckin__add_monitor__add_index_monitor_type_next_che
 > sentry:0458_global_searches_data_migration
Saved Searchs: 100% |########################################################################################################################################################################| Time: 0:00:07
 > sentry:0459_global_searches_unique_constraint
 > sentry:0460_auto__add_field_servicehook_organization_id
 > sentry:0461_event_attachment_indexes
 > sentry:0462_auto__add_servicehookproject
 > sentry:0462_releaseenvironment_project_id
 > sentry:0463_backfill_service_hook_project
 > sentry:0464_auto__add_sentryappcomponent__add_field_sentryapp_schema
 > sentry:0464_groupenvironment_foreignkeys
 > sentry:0465_sync
 > sentry:0466_auto__add_platformexternalissue__add_unique_platformexternalissue_grou
 > sentry:0467_backfill_integration_status
 > sentry:0468_auto__add_field_projectdebugfile_code_id__add_index_projectdebugfile_p
 > sentry:0468_recent_search
 > sentry:0469_fix_state
 > sentry:0470_org_saved_search
 > sentry:0471_global_saved_search_types
 > sentry:0472_auto__add_field_sentryapp_author
The following content types are stale and need to be deleted:

    sentry | dsymobject
    sentry | dsymapp
    sentry | useridentity
    sentry | dsymsymbol
    sentry | dsymsdk
    sentry | globaldsymfile
    sentry | versiondsymfile
    sentry | projectdsymfile
    sentry | grouphashtombstone
    sentry | minidumpfile
    sentry | dsymbundle

Any objects related to these content types by a foreign key will also
be deleted. Are you sure you want to delete these content types?
If you're unsure, answer 'no'.

    Type 'yes' to continue, or 'no' to cancel: yes
Running migrations for sentry.nodestore:
- Nothing to migrate.
Running migrations for sentry.search:
- Nothing to migrate.
Running migrations for social_auth:
- Nothing to migrate.
Running migrations for sentry.tagstore:
- Nothing to migrate.
Running migrations for sentry_plugins.hipchat_ac:
- Nothing to migrate.
Running migrations for sentry_plugins.jira_ac:
- Nothing to migrate.

Synced:
 > django.contrib.admin
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.messages
 > django.contrib.sessions
 > django.contrib.sites
 > django.contrib.staticfiles
 > crispy_forms
 > debug_toolbar
 > rest_framework
 > sentry.plugins.sentry_interface_types
 > sentry.plugins.sentry_mail
 > sentry.plugins.sentry_urls
 > sentry.plugins.sentry_useragents
 > sentry.plugins.sentry_webhooks
 > sudo
 > south

Migrated:
 - sentry
 - sentry.nodestore
 - sentry.search
 - social_auth
 - sentry.tagstore
 - sentry_plugins.hipchat_ac
 - sentry_plugins.jira_ac
Creating missing DSNs
Correcting Group.num_comments counter
Cleaning up...

----------------
You're all done! Run the following command get Sentry running:

  docker-compose up -d




É isso, a versão 9.1.2 está instalada.



Conclusões.



Essas são verdades comuns conhecidas por todos. Não tenha pressa, faça backups e verifique tudo várias vezes.



Obrigado pela atenção!


More articles:


All Articles