Roteiro do Odyssey: o que mais queremos do extrator de conexão. Andrey Borodin

O relatório Andrei Borodin conta como eles levaram em consideração a experiência de dimensionamento do PgBouncer no projeto do extrator de compostos Odyssey à medida que ele entrava em produção. Além disso, discutiremos quais funções do extrator gostaríamos de ver em novas versões: é importante para nós não apenas cobrir nossas necessidades, mas desenvolver a comunidade de usuários Odyssey .

Vídeo:

! .

open source . connection pooler .

, connection pooler -, . , .

, connection pooler . - , , , Postgres . ( HighLoad++ 2019) Postgres. , , , , , , . , .

Postgres. . . , Postgres.

– primary , ( ), , .

– Postgres, Postgres connection pooler. Connection pooler fencing .

connection pooler?

Postgres . , – , Postgres. , .

, Postgres , procArray. . procArray , . , . , .

3 :

• .
• .
• , . . .

, pooler . PostgreSQL Professional . , . . application-side pool proxy pool.

Application-side pool – . : .

, , .

, , -. client side pooling . – 10 000 . , .

proxy poolers, poolers, . poolers. poolers + . Pgpool Crunchy-Proxy.

, , . , poolers pooling, . . , .

, PgBouncer, transaction pooling, . . .

– . .

, , . loopback - .

, application_name_add_host. Bouncer IP- application_name. application_name .

, – , – , . – application_name, , .

, Bouncer pool, . . , .

? , C++ - , , . 20 000 , . .

, , Bouncer, , . . pool.

Postgres, . . .

, . PgBouncer , . : , , , , , - . . , , .

, .

, Bouncer . . , , pooler.

, PgBouncer’.

https://lwn.net/Articles/542629/

Bouncer.

, TCP- Bouncers. TCP- round-robin’ .

, . . , Bouncer, idle- Bouncer’.

, 3 Bouncers 100 %. Bouncers. ?

TLS. . Postgres TLS TLS, , , TLS handshake .

, . primary , – . . . - primary , -. TLS.

TLS handshake Bouncer, . - . retry exponential backoff, .

16 PgBouncer, 16 100 %.

PgBouncer. , Bouncer. Bouncers TCP handshake, Bouncers pooling, , .

. 18 Bouncers . . , DevOps , , .

, open source, Bouncer . , , PgBouncers . pull request .

https://www.postgresql.org/docs/current/libpq-cancel.html

https://github.com/pgbouncer/pgbouncer/pull/79

. Postgres , - . TCP-reset, . . . Bouncer? . . , , Bouncer , , .

upstream Bouncer’.

, connection pooler, , , , , .

. TLS-.

, Machinarium, . libpq, , : « . IO, , IO ». . . " N, N ", " SYNC, ". , . , . , , (green thread-). , , .

, TCP accept round-robin' workers TPC-.

. cache-friendly.

, , TCP-stack.

, , Odyssey CANCEL ROLLBACK , . . , Odyssey , , .

. application_name_add_host. , , .

.. managed PostgreSQL connection pooler, , . . , , . Bouncer .

.

, . , , , . , connection pooler Odyssey, .

Odyssey PgBouncer. , . - , pull request, issue GitHub, . PgBouncer .

, , error forwarding. , . , , , .

, 100%- PgBouncer. , Bouncer, .

Odyssey.

https://github.com/yandex/odyssey/pull/66

, «Pause /Resume». . Postgres, connection pooler, pg_upgrade, resume. , . . , . ( )

https://github.com/yandex/odyssey/pull/73 —

, PgBouncer, SCRAM Authentication, , .. – .

Odyssey, .

Odyssey, . Kiwi – Postgres’ . . . proto 3 Postgres – , , . Kiwi.

Machinarium – . Machinarium . , , 15 .

Odyssey. , coroutines. accept TCP- workers.

worker . crone- , pool.

Odyssey Postgres. install-check Bouncer Odyssey, div. , Bouncer Odyssey.

, , . Odyssey.

, - : Postgres + Odyssey, PgBouncer + Odyssey, Odyssey + Odyssey , , , Odyssey - , - , .

Odyssey production. , , . , . . , . , production , PostgreSQL Professional , memory leak. , . .

, connection pooler TLS- TLS-. .

Bouncer Odyssey pcache, pcache , Odyssey . , . , 20 000 .

Pluggable Authentication Method – lunux’ . PgBouncer , PAM PgBouncer, PAM.

. . ?

, , PAM- PAM-, PAM- PAM-. , . , .

, , accept . worker pool, TLS handshake.

, 20 000 , . libpq . 3 .

, , retry.

, PgBouncer , throttling TCP-, accept.

, , , , . , handshake , . - , .

Roadmap

Odyssey? ?

2019 .

roadmap Odyssey :

• SCRAM PAM authentication.
• forward standby.
• online-restart.
• .

roadmap , . . , .

forward read-only queries to standby? , . failover switchover. - - . , -, .

, Postgres, 10- session_attrs. , : . , , session_attrs.

, . - , . , , Odyssey , .

Odyssey primary. , , , , . .

, open source. , , 2,5 PgBouncer. Odyssey.

prepared statement. prepared statement . -, SQL-, «prepared». SQL- SQL Bouncer. overkill, , . . . SQL-.

prepared statement proto3. , , prepared statement, . , - prepared statements. , - .

, - , , prepared statements , , . . prepared statement.

, , prepared statement, . , , , , , , , .

, . , PgBouncer. . – : - , - – . .

, , , , .

– 1.0 ( 1.1). , Odyssey 1.0rc, . . release candidate. , , , memory leak.

1.0? Odyssey . , 1 000 000 , , , 1.0.

, 1.0 SCRAM. , production , SCRAM, . , , .

pull request. , Bouncer. . , - , .

, . !

application_name, , transaction pooling Odyssey?

Odyssey Bouncer?

Odyssey. Bouncer .

.

, ?

, . , application_name. , . . , .

, , ! ! , Odyssey . . multi data-source , Odyssey , . . slave, failover .

, , , . storages. . , , failover, pg_recovery. , . - ? . . , . . , C, .

, adoption . , . . , .

. , . , , , . , , , . , , , , , . . . , , .

, . pcache , , real cache , , , .

- , , , , . . .

. , . , , , 10 . , .

, .

. . , , . , .

O Nginx tem essa opção slowly startno cluster para o servidor. E ele aumenta gradualmente a carga.

Sim, ótima ideia, vamos tentar quando chegarmos lá.