无法扩展使用 sqlalchemy 和 postgres 开发的 python 后端应用程序应用程序未使用 postgres db

如何解决无法扩展使用 sqlalchemy 和 postgres 开发的 python 后端应用程序应用程序未使用 postgres db

我正在使用 FastAPI、sqlalchemy 和 Postgres 数据库开发后端应用程序。我正在使用以下脚本创建会话

engine = create_engine(
    f"postgresql://{POSTGRES_USER}:{POSTGRES_PASSWORD}@{POSTGRES_HOST}:{POSTGRES_PORT}/{POSTGRES_DB}",pool_pre_ping=True,echo=False,pool_size=20,max_overflow=30,)

SessionLocal = sessionmaker(autocommit=False,autoflush=False)


def get_session():
    conn = engine.connect()
    session = SessionLocal(bind=conn)
    try:
        yield session
    finally:
        session.close()

无论何时进行 API 调用，都会触发 get_session 依赖项，如 fastAPI 文档中所述https://fastapi.tiangolo.com/tutorial/sql-databases/#create-a-dependency

我的 Postgress 以最大连接数 = 200 运行

当我使用 jMeter 测试我的应用程序时，100 个用户和每个用户 10 个请求，我的应用程序失败并出现以下错误。

2021-02-24 19:44:20,825 ERROR    uvicorn.error        Exception in ASGI application
Traceback (most recent call last):
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py",line 390,in run_asgi
    result = await app(self.scope,self.receive,self.send)
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py",line 45,in __call__
    return await self.app(scope,receive,send)
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/fastapi/applications.py",line 181,in __call__
    await super().__call__(scope,send)
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/starlette/applications.py",line 111,in __call__
    await self.middleware_stack(scope,send)
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/starlette/middleware/errors.py",in __call__
    raise exc from None
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/starlette/middleware/errors.py",line 159,in __call__
    await self.app(scope,_send)
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/starlette/middleware/cors.py",line 78,send)
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/starlette/exceptions.py",line 82,in __call__
    raise exc from None
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/starlette/exceptions.py",line 71,sender)
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/starlette/routing.py",line 566,in __call__
    await route.handle(scope,send)
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/starlette/routing.py",line 227,in handle
    await self.app(scope,line 41,in app
    response = await func(request)
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/fastapi/routing.py",line 190,in app
    dependency_overrides_provider=dependency_overrides_provider,File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/fastapi/dependencies/utils.py",line 536,in solve_dependencies
    call=call,stack=stack,sub_values=sub_values
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/fastapi/dependencies/utils.py",line 453,in solve_generator
    return await stack.enter_async_context(cm)
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/contextlib.py",line 570,in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/contextlib.py",line 170,in __aenter__
    return await self.gen.__anext__()
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/site-packages/fastapi/concurrency.py",in contextmanager_in_threadpool
    raise e
  File "/Users/rahul/opt/anaconda3/envs/random_project/lib/python3.7/contextlib.py",line 130,in __exit__
    self.gen.throw(type,value,traceback)
sqlalchemy.exc.TimeoutError: QueuePool limit of size 20 overflow 30 reached,connection timed out,timeout 30 (Background on this error at: http://sqlalche.me/e/13/3o7r)

当我在高峰期检查我的数据库的活动连接数时，它如下

应用未运行时的空闲时间。

datacat=# select max_conn,used,res_for_super,max_conn-used-res_for_super res_for_normal
from
  (select count(*) used from pg_stat_activity) t1,(select setting::int res_for_super from pg_settings where name=$$superuser_reserved_connections$$) t2,(select setting::int max_conn from pg_settings where name=$$max_connections$$) t3
;
 max_conn | used | res_for_super | res_for_normal
----------+------+---------------+----------------
      200 |    6 |             3 |            191
(1 row)

当我的应用程序开始因上述错误而失败时。我的 Python 应用程序没有建立额外的连接来满足请求，它只有 56 个活动连接。其中 50 个是在运行后端时创建的新连接。

 max_conn | used | res_for_super | res_for_normal
----------+------+---------------+----------------
      200 |   56 |             3 |            141

1. 我该怎么做才能提高性能并修复上述错误。
2. 根据我的理解，首先创建一个连接，池大小为 20，max_overflow 为 30。然后当创建会话时，会话将尽可能从该池中获取一个连接。如果此池已满，将创建一个大小为 20 的新连接，我将从该池中获取连接。当我完成我的手术时。我将关闭我的会话，并让连接保持打开状态，因为它可以被另一个正在等待的会话使用。我对吗？我很困惑，因为对 this link 的回答说我应该关闭 conn 而不是 session。如果是这样，为什么 fastAPI 文档说关闭会话？
3. 即使我设置 max_overflow=0 以便在同一个池中没有等待连接的请求，我也会收到错误 sqlalchemy.exc.TimeoutError: QueuePool limit of size 20 overflow 0 reached,timeout 30。什么是超时？我的查询已经在池中获得了连接？或者是其他东西。我不认为这是我的查询，因为平均我的查询需要
4. 在上面的场景中（数字 3，pool_size=20，max_overflow=0）如果有一个有 20 个连接的池，那么其他连接请求会怎样？他们在哪儿等？