[{"content":"Async tests and fixtures Pytest already has the official pytest-asyncio plugin which allows you to write async tests and fixtures.\nAccording to pytest-asyncio docs, you can have an async test with an async fixture like this:\nfrom typing import Any, AsyncGenerator import pytest import pytest_asyncio from httpx import AsyncClient @pytest_asyncio.fixture async def client() -\u0026gt; AsyncGenerator[AsyncClient, Any]: async with AsyncClient() as c: yield c @pytest.mark.asyncio async def test_api_call(client: AsyncClient) -\u0026gt; None: response = await client.get(\u0026#34;http://test.com\u0026#34;) assert response.status_code == 200 There are a few points to consider here:\nObviously the fixture and test are defined as async def since that was the intention. The fixture should be decorated with @pytest_asyncio.fixture so pytest can pick this up properly. The tests should be marked with @pytest.mark.asyncio decorator. As an alternative you can define a variable pytestmark = pytest.mark.asyncio in your test file to treat all tests as async and avoid the repetition.\nDiscovery mode This works well but with some tweaks it can be made simpler. The pytest-asyncio offers a discovery mode concept which allows you to control how async tests are discovered.\nIf your project only uses asyncio as the asynchronous programming library, you can take advantage of the discovery mode to make this simpler:\nfrom typing import Any, AsyncGenerator import pytest from httpx import AsyncClient @pytest.fixture async def client() -\u0026gt; AsyncGenerator[AsyncClient, Any]: async with AsyncClient() as c: yield c async def test_api_call(client: AsyncClient) -\u0026gt; None: response = await client.get(\u0026#34;http://test.com\u0026#34;) assert response.status_code == 200 This way:\nYou can define fixtures with normal @pytest.fixture decorator. You don\u0026rsquo;t need to mark tests as async, so no decorators needed. You can now run the tests with:\n$ pytest tests --asyncio-mode=auto If you are using pyproject.toml file you can add this with:\n[tool.pytest.ini_options] asyncio_mode = \u0026#34;auto\u0026#34; And run the tests with:\n$ pytest tests Asyncio event loop When testing a FastAPI or SQLAlchemy project, you might get the error \u0026lt;Task pending\u0026gt; attached to a different loop.\nThe reason is that you should define event loop for pytest to use. You can add the event_loop fixture to conftest.py with:\nimport asyncio from typing import Any, Generator import pytest @pytest.fixture(scope=\u0026#34;session\u0026#34;) def event_loop() -\u0026gt; Generator[asyncio.AbstractEventLoop, Any, None]: loop = asyncio.get_event_loop() yield loop loop.close() ","permalink":"https://aminalaee.github.io/posts/2023/pytest-async-tests/","summary":"Async tests and fixtures Pytest already has the official pytest-asyncio plugin which allows you to write async tests and fixtures.\nAccording to pytest-asyncio docs, you can have an async test with an async fixture like this:\nfrom typing import Any, AsyncGenerator import pytest import pytest_asyncio from httpx import AsyncClient @pytest_asyncio.fixture async def client() -\u0026gt; AsyncGenerator[AsyncClient, Any]: async with AsyncClient() as c: yield c @pytest.mark.asyncio async def test_api_call(client: AsyncClient) -\u0026gt; None: response = await client.","title":"Pytest async tests and fixtures"},{"content":"Summary Recently I was working on a project that integrated with some internal and external APIs using HTTP requests. You probably have heard about or worked with the requests library in Python which is probably the de-facto HTTP client package which is much easier to work with compared to the built-in HTTP module of Python.\nThe previous implementation in our project was using the requests library and for each new request it did something like requests.get(...) to do the integration with other services.\nIf you have worked a bit with requests you probably know about the Session object or it\u0026rsquo;s equivalent Client object in httpx. This reminded me of this section in Python httpx documentation:\nIf you do anything more than experimentation, one-off scripts, or prototypes, then you should use a Client instance.\nThe benefit of using Session or Client objects is that you will be able to do HTTP persistent connections, and if that\u0026rsquo;s supported by the server too, the underlying TCP connection between your client and server will be kept open, so you will re-use the same resources for future requests.\nUsage Let\u0026rsquo;s write a simple script to see how connection pooling works in action. This script will just enable DEBUG logging and send some requests to httpbin.org.\nLet\u0026rsquo;s send some requests in a naive way:\nimport logging import requests logging.basicConfig(level=logging.DEBUG) requests.get(\u0026#34;http://httpbin.org\u0026#34;) requests.get(\u0026#34;http://httpbin.org\u0026#34;) And this is the log generated:\nDEBUG:urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org:80 DEBUG:urllib3.connectionpool:http://httpbin.org:80 \u0026#34;GET / HTTP/1.1\u0026#34; 200 9593 DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org:80 DEBUG:urllib3.connectionpool:http://httpbin.org:80 \u0026#34;GET / HTTP/1.1\u0026#34; 200 9593 As expected a new HTTP connection is started per request. Now let\u0026rsquo;s try to use a Session object to see the difference.\nimport logging import requests logging.basicConfig(level=logging.DEBUG) session = requests.Session() session.get(\u0026#34;http://httpbin.org\u0026#34;) session.get(\u0026#34;http://httpbin.org\u0026#34;) And now running this you will see something like:\nDEBUG:urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org:80 DEBUG:urllib3.connectionpool:http://httpbin.org:80 \u0026#34;GET / HTTP/1.1\u0026#34; 200 9593 DEBUG:urllib3.connectionpool:http://httpbin.org:80 \u0026#34;GET / HTTP/1.1\u0026#34; 200 9593 Great, for the first HTTP request a new HTTP connection is opened, but for the second request no new connection is opened, thanks to the urllib3\u0026rsquo;s connection pool.\nBenchmarking For the setup I\u0026rsquo;m running a dummy Django application with gunicorn and using a Caddy as reverse proxy to handle the persistent connections.\nSo here I start the Django application:\n$ gunicorn example.wsgi -w 4 And in another terminal, run reverse proxy:\n$ caddy reverse-proxy --from :9000 --to :8000 And now let\u0026rsquo;s run a simple script to benchmark the difference:\nimport requests url = \u0026#34;http://localhost:9000/\u0026#34; def time_requests(): elapsed = 0 for _ in range(1_000): response = requests.get(url) elapsed += response.elapsed.total_seconds() print(f\u0026#34;time_requests --\u0026gt; elapsed: {elapsed}\u0026#34;) def time_requests_session(): session = requests.Session() elapsed = 0 for _ in range(1_000): response = session.get(url) elapsed += response.elapsed.total_seconds() print(f\u0026#34;time_requests_session --\u0026gt; elapsed: {elapsed}\u0026#34;) time_requests() time_requests_session() This is the result I get on my laptop, but you should probably get the same results:\ntime_requests --\u0026gt; elapsed: 2.1866910000000015 time_requests_session --\u0026gt; elapsed: 1.7474469999999986 So we could easily reach 25% of performance improvement with literally adding one line of code. Of course this will vary based on your use-case and environment, but generally it should be an improvement compared to the previous approach.\nNow let\u0026rsquo;s do the same thing with httpx:\nimport httpx def time_httpx_client(): client = httpx.Client() elapsed = 0 for _ in range(1_000): response = client.get(url) elapsed += response.elapsed.total_seconds() print(f\u0026#34;time_httpx_client --\u0026gt; elapsed: {elapsed}\u0026#34;) time_httpx_client() Which gives me the same result as requests.Session approach. The nice thing about httpx is that even though it supports both sync and async usage, most of the interface is similar to the requests.\nFinal notes You might be wondering why I\u0026rsquo;m using response.elapsed.total_seconds() in both scripts. We could just get the timestamps before and after the loop and have the total time calculated.\nI think the requests docs explains elapsed attribute very well:\nThe amount of time elapsed between sending the request and the arrival of the response (as a timedelta). This property specifically measures the time taken between sending the first byte of the request and finishing parsing the headers. It is therefore unaffected by consuming the response content or the value of the stream keyword argument.\nSo this will (hopefully) give a more realistic benchmark that we are not including the response body consumption into our benchmark, but remember benchmarks can be misleading and you need to test it for your own use-case to see how it works!\n","permalink":"https://aminalaee.github.io/posts/2023/python-requests-persistent-connections/","summary":"Summary Recently I was working on a project that integrated with some internal and external APIs using HTTP requests. You probably have heard about or worked with the requests library in Python which is probably the de-facto HTTP client package which is much easier to work with compared to the built-in HTTP module of Python.\nThe previous implementation in our project was using the requests library and for each new request it did something like requests.","title":"Python requests Connection Pool"},{"content":"Intro If you are already familiar with Pydantic, one of the useful components of Pydantic is the Settings Management. This will allow you to read settings variables from different sources and parse and validate them into class(es) using Pydantic.\nLet\u0026rsquo;s see a minimal example. First we need to set up the variable:\n$ export API_KEY=xxx And then we can read it into the Settings class with:\nfrom pydantic import BaseSettings class Settings(BaseSettings): api_key: str print(Settings().dict()) \u0026#34;\u0026#34;\u0026#34; {\u0026#39;api_key\u0026#39;: \u0026#39;xxx\u0026#39;} \u0026#34;\u0026#34;\u0026#34; Different data sources are available to read variables from, currently the options include:\nEnvironment variables Dotenv file Docker secrets But what if you want to add new data sources? For example if you want to read it from a JSON file or AWS Secrets Manager?\nLuckily Pydantic Settings allows you to customize the settings resources easily. Checking here you can add or remove resources or change their priorities.\nSo let\u0026rsquo;s see how we can add a custom resource to read secrets from AWS Secrets Manager.\nAWS Secrets Manager Note: If you are familiar with AWS Secrets Manager or already have access to an existing one with some secrets defined you can skip to the next section.\nSecrets Manager is a service which allows you to store any sensitive information like database passwords, API Keys, etc and retrieve them in your code without worrying about environment variables or dotenv files being passed around in your source code. Obviously your secrets are encrypted in AWS and decrypted in two steps when you want to retrieve them.\nOn top of this, you also get other benefits like secret rotation, logging and managing permissions based on AWS IAM policies.\nIf you already have an AWS account and credentials you can move to the next step and use the example code, otherwise I will use localstack to create a local version of all AWS services and test this without actually working with an AWS account.\nYou can install localstack easily with:\n$ pip install localstack After it\u0026rsquo;s installed, you can start localstack to run using Docker:\n$ localstack start -d AWS services should now be available on your local at http://localhost:4566.\nIf you alrady have aws CLI installed, you can try using the localstack with:\n$ export AWS_ACCESS_KEY_ID=\u0026#34;test\u0026#34; $ export AWS_SECRET_ACCESS_KEY=\u0026#34;test\u0026#34; $ export AWS_DEFAULT_REGION=\u0026#34;us-east-1\u0026#34; $ aws --endpoint-url=http://localhost:4566 secretsmanager list-secrets Or you can install awslocal, which is a wrapper around aws CLI and you don\u0026rsquo;t need to use the --endpoint-url anymore.\n$ pip install awscli-local $ awslocal secretsmanager list-secrets This will probably get you some empty response because we haven\u0026rsquo;t defined any secrets yet.\nNow let\u0026rsquo;s try to define two secrets:\n$ awslocal secretsmanager create-secret \\ --name example_secret \\ --secret-string \u0026#34;SECRET_VALUE\u0026#34; $ awslocal secretsmanager create-secret \\ --name example_complex \\ --secret-string \u0026#34;{\\\u0026#34;user\\\u0026#34;:\\\u0026#34;diegor\\\u0026#34;,\\\u0026#34;password\\\u0026#34;:\\\u0026#34;EXAMPLE-PASSWORD\\\u0026#34;}\u0026#34; Next let\u0026rsquo;s try to list the secrets again with awslocal secretsmanager list-secrets which will give a similiar response to this:\n{ \u0026#34;SecretList\u0026#34;: [ { \u0026#34;ARN\u0026#34;: \u0026#34;arn:aws:secretsmanager:us-east-1:000000000000:secret:example_secret-xxxx\u0026#34;, \u0026#34;Name\u0026#34;: \u0026#34;example_secret\u0026#34;, \u0026#34;LastChangedDate\u0026#34;: xxxxxxxxxxxxxxxxx, \u0026#34;SecretVersionsToStages\u0026#34;: { \u0026#34;e49ae6a9-201b-45a8-93ea-a474cf3427cb\u0026#34;: [ \u0026#34;AWSCURRENT\u0026#34; ] }, \u0026#34;CreatedDate\u0026#34;: xxxxxxxxxxxxxxxxx }, { \u0026#34;ARN\u0026#34;: \u0026#34;arn:aws:secretsmanager:us-east-1:000000000000:secret:example_complex-xxxx\u0026#34;, \u0026#34;Name\u0026#34;: \u0026#34;example_complex\u0026#34;, \u0026#34;LastChangedDate\u0026#34;: xxxxxxxxxxxxxxxxx, \u0026#34;SecretVersionsToStages\u0026#34;: { \u0026#34;87b93858-f03a-431e-8560-2c1ec836945f\u0026#34;: [ \u0026#34;AWSCURRENT\u0026#34; ] }, \u0026#34;CreatedDate\u0026#34;: xxxxxxxxxxxxxxxxx } ] } Now we can move to our implementation code.\nPydantic Settings with Secrets Manager The first step is to connect to the AWS secrets manager service using boto3 and create a client. In case of using localstack you can use the following credentials to connect to it:\nfrom boto3 import Session session = Session( aws_access_key_id=\u0026#34;test\u0026#34;, aws_secret_access_key=\u0026#34;test\u0026#34;, region_name=\u0026#34;us-east-1\u0026#34;, ) client = session.client( endpoint_url=\u0026#34;http://localhost:4566\u0026#34;, service_name=\u0026#34;secretsmanager\u0026#34;, region_name=\u0026#34;us-east-1\u0026#34;, ) Pydantic BaseSettings class can use a nested Config class which does extra configurations for the settings management. Next we define a SecretManagerConfig class which will modify Config\u0026rsquo;s behaviour:\nimport json from typing import Any from pydantic.env_settings import SettingsSourceCallable class SecretManagerConfig: @classmethod def _get_secret(cls, secret_name: str) -\u0026gt; str | dict[str, Any]: # Get secret string value secret_string = client.get_secret_value(SecretId=secret_name)[\u0026#34;SecretString\u0026#34;] try: # try to decode it into a dict if it was JSON encoded return json.loads(secret_string) except json.decoder.JSONDecodeError: return secret_string @classmethod def get_secrets(cls, settings: BaseSettings) -\u0026gt; dict[str, Any]: # We go through the fields and try to fetch a secret with that field name return { name: cls._get_secret(name) for name, _ in settings.__fields__.items() } @classmethod def customise_sources( cls, init_settings: SettingsSourceCallable, env_settings: SettingsSourceCallable, file_secret_settings: SettingsSourceCallable, ): # Here we add the `cls.get_secrets` method as an extra data source return ( init_settings, cls.get_secrets, env_settings, file_secret_settings, ) The method customise_sources is the method that Pydantic Settings allows you to add or delete new data sources. So it\u0026rsquo;s overriden to add our custom data source get_secrets there.\nIn get_secrets we go through the Settings.__fields__ and get the secret details by using this name. For each secret string returned, we can try to load it into a dict using json.loads in case our secret was JSON encoded.\nAnd we define our settings classes next:\nfrom pydantic import BaseSettings, BaseModel class DatabaseSettings(BaseModel): user: str password: str class Settings(BaseSettings): example_secret: str example_complex: DatabaseSettings class Config(SecretManagerConfig): ... And finally we can load our settings with:\nprint(Settings().dict()) \u0026#34;\u0026#34;\u0026#34; {\u0026#39;example_secret\u0026#39;: \u0026#39;SECRET_VALUE\u0026#39;, \u0026#39;example_complex\u0026#39;: {\u0026#39;user\u0026#39;: \u0026#39;diegor\u0026#39;, \u0026#39;password\u0026#39;: \u0026#39;EXAMPLE-PASSWORD\u0026#39;}} \u0026#34;\u0026#34;\u0026#34; This allows us to load both simple key, value secrets and JSON encoded ones into sub-settings which Pydantic handles very nicely.\nObviously this can be improved to handle more cases or even be written in better ways, but this will give you the idea how you can do continue from here.\nThe complete code:\nimport json from typing import Any from boto3 import Session from pydantic import BaseSettings, BaseModel from pydantic.env_settings import SettingsSourceCallable session = Session( aws_access_key_id=\u0026#34;test\u0026#34;, aws_secret_access_key=\u0026#34;test\u0026#34;, region_name=\u0026#34;us-east-1\u0026#34;, ) client = session.client( endpoint_url=\u0026#34;http://localhost:4566\u0026#34;, service_name=\u0026#34;secretsmanager\u0026#34;, region_name=\u0026#34;us-east-1\u0026#34;, ) class SecretManagerConfig: @classmethod def _get_secret(cls, secret_name: str) -\u0026gt; str | dict[str, Any]: secret_string = client.get_secret_value(SecretId=secret_name)[\u0026#34;SecretString\u0026#34;] try: return json.loads(secret_string) except json.decoder.JSONDecodeError: return secret_string @classmethod def get_secrets(cls, settings: BaseSettings) -\u0026gt; dict[str, Any]: return { name: cls._get_secret(name) for name, _ in settings.__fields__.items() } @classmethod def customise_sources( cls, init_settings: SettingsSourceCallable, env_settings: SettingsSourceCallable, file_secret_settings: SettingsSourceCallable, ): return ( init_settings, cls.get_secrets, env_settings, file_secret_settings, ) class DatabaseSettings(BaseModel): user: str password: str class Settings(BaseSettings): example_secret: str example_complex: DatabaseSettings class Config(SecretManagerConfig): ... print(Settings().dict()) \u0026#34;\u0026#34;\u0026#34; {\u0026#39;example_secret\u0026#39;: \u0026#39;SECRET_VALUE\u0026#39;, \u0026#39;example_complex\u0026#39;: {\u0026#39;user\u0026#39;: \u0026#39;diegor\u0026#39;, \u0026#39;password\u0026#39;: \u0026#39;EXAMPLE-PASSWORD\u0026#39;}} \u0026#34;\u0026#34;\u0026#34; ","permalink":"https://aminalaee.github.io/posts/2023/pydantic-settings-secret-manager/","summary":"Intro If you are already familiar with Pydantic, one of the useful components of Pydantic is the Settings Management. This will allow you to read settings variables from different sources and parse and validate them into class(es) using Pydantic.\nLet\u0026rsquo;s see a minimal example. First we need to set up the variable:\n$ export API_KEY=xxx And then we can read it into the Settings class with:\nfrom pydantic import BaseSettings class Settings(BaseSettings): api_key: str print(Settings().","title":"Pydantic Settings with AWS Secrets Manager"},{"content":"Intro From AWS website:\nAWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers.\nEven though it\u0026rsquo;s not required to use a web framework like FastAPI or Django to start using AWS Lambda, often it is the case you have an existing project or you want to use a framework for useful features.\nThe Mangum project fixes the gap between the existing ASGI Python frameworks to work with AWS Lambda serverless functions. For the web framework I will be using FastAPI but the same can be achieved with Starlette applications.\nAWS provides a few options for the deployment of Lambda functions, the simplest one which I will use is to create a runtime environment and package a Zip from it and send it to AWS. By runtime it means you will upload an archive which has all the dependencies and the logic you want for your Lambda. In real projects of course this is not ideal and you will probably use container images.\nSimple example First we create a virtual environment and install the required packages:\n$ python3.9 -m venv venv $ source venv/bin/activate $ pip install fastapi mangum And we will create a Zip archive from it:\n$ cd venv/lib/python3.9/site-packages $ zip -r ../../../../package.zip . This will zip the virtual environment to be uploaded to AWS.\nNext we create a minimal API in the file example.py:\nfrom fastapi import FastAPI from mangum import Mangum app = FastAPI() @app.get(\u0026#34;/\u0026#34;) def root(): return {\u0026#34;message\u0026#34;: \u0026#34;Up and running!\u0026#34;} handler = Mangum(app) As you can see only the last line is extra and that is how AWS will trigger our Lambda function.\nNext we cd back into our root of project and add example.py to the archive too:\n$ cd ../../../../ $ zip -g package.zip example.py Now our archive can be uploaded to AWS. We can either do this with AWS CLI or console.\nAfter our zip is uploaded we need to update the Runtime settings of the function, so that AWS can locate the handler. In the case of this example the handler should be set to example.handler meaning the in the example.py file it should look for a variable called handler.\nDone!\nYour function is deployed. Now you can go to the configuration tab of the function and create a Function URL which is a public/private URL you can use to test your API.\nLarger applications Even though this is already great, there\u0026rsquo;s a downside:\nIn real-world projects it would not be efficient to deploy the whole application in one Function. As the Lambda Function scales up and down automatically, you might want to scale one endpoint but not the other endpoints. If your app is deployed in one monolithic lambda it will scale up/down all together:\nInstead, it is more efficient to deploy each group of endpoints in one Lambda so you can get the most from serverless architecture and let the API Gateway manage the routing for you:\nSo how can we achieve this?\nFastAPI offers APIRouter, or if you are familiar with Starlette you have Router instances. Basically APIRouter or Router is a lightweight ASGI application without the middlewares, but since they are like ASGI apps you can use them with Mangum.\nNow create another file complex.py:\nfrom fastapi import FastAPI, APIRouter from mangum import Mangum app = FastAPI() router_foo = APIRouter() router_bar = APIRouter() @router_foo.get(\u0026#34;/foo\u0026#34;) def foo(): return {\u0026#34;message\u0026#34;: \u0026#34;Hello Foo!\u0026#34;} @router_bar.get(\u0026#34;/bar\u0026#34;) def bar(): return {\u0026#34;message\u0026#34;: \u0026#34;Hello Bar!\u0026#34;} app.include_router(router_foo) app.include_router(router_bar) handler_foo = Mangum(router_foo) handler_bar = Mangum(router_bar) Note here that we passed an APIRouter instance to Mangum(...), instead of the application.\nNow we can repeate the process to add the handlers to the zip file:\n$ zip -g package.zip complex.py Next we go to the AWS console and create two different functions for our handlers. One Function should have the handler complex.handler_foo and another complex.handler_bar.\nDone!\nNow Each one of our APIRouters will be scaled differently. Again you can give the Function a URL in configurations and test your deployment.\nReferences: AWS blog post here about the best practices with Lambda. AWS tutorial about Python Zip deployments for Lambda here. ","permalink":"https://aminalaee.github.io/posts/2022/fastapi-aws-lambda/","summary":"Intro From AWS website:\nAWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers.\nEven though it\u0026rsquo;s not required to use a web framework like FastAPI or Django to start using AWS Lambda, often it is the case you have an existing project or you want to use a framework for useful features.","title":"FastAPI AWS Lambda deployment"},{"content":"Use-case It\u0026rsquo;s very common for projects to do JSON logging if you are working with third-party tools or open-source projects like Logstash to process your logs. These tools usually need more complex filtering on the structured data, so using JSON is preferred there.\nWe also wanted to integrate with a third-party tool at work and we needed to add the JSON formatting logs in our projects.\nI will not go into the details of why or why not you should decide JSON logging, as each approach will have it\u0026rsquo;s pros and cons. Instead I will explain how you can do it in Python.\nThe final goal would be to go from:\n2022-09-14 23:47:11,506248 - myapp - DEBUG - debug message To:\n{ \u0026#34;threadName\u0026#34;: \u0026#34;MainThread\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;root\u0026#34;, \u0026#34;thread\u0026#34;: 140735202359648, \u0026#34;created\u0026#34;: 1336281068.506248, \u0026#34;process\u0026#34;: 41937, \u0026#34;processName\u0026#34;: \u0026#34;MainProcess\u0026#34;, \u0026#34;relativeCreated\u0026#34;: 9.100914001464844, \u0026#34;module\u0026#34;: \u0026#34;app\u0026#34;, \u0026#34;funcName\u0026#34;: \u0026#34;do_logging\u0026#34;, \u0026#34;levelno\u0026#34;: 20, \u0026#34;pathname\u0026#34;: \u0026#34;app.py\u0026#34;, \u0026#34;lineno\u0026#34;: 20, \u0026#34;asctime\u0026#34;: [\u0026#34;2022-09-14 23:47:11,506248\u0026#34;], \u0026#34;message\u0026#34;: \u0026#34;debug message\u0026#34;, \u0026#34;filename\u0026#34;: \u0026#34;main.py\u0026#34;, \u0026#34;levelname\u0026#34;: \u0026#34;DEBUG\u0026#34;, } Existing projects If you do a quick search, like I did, you will find two (more or less) active projects which do this:\npython-json-logger json-log-formatter And more And it\u0026rsquo;s interesting that if you check the downloads of these project at PePY or some other tool you see many people probably actually use them. As of writing this, I checked that python-json-logger has a daily download rate of ~200K per day!\nWhy I think you probably don\u0026rsquo;t need that The Python logging module provides a Formatter which can be used to do logging in any formatting you want.\nA very simple and minimal example of a JSON formatter can be written as:\nimport json import logging class JSONFormatter(logging.Formatter): def __init__(self) -\u0026gt; None: super().__init__() self._ignore_keys = {\u0026#34;msg\u0026#34;, \u0026#34;args\u0026#34;} def format(self, record: logging.LogRecord) -\u0026gt; str: message = record.__dict__.copy() message[\u0026#34;message\u0026#34;] = record.getMessage() for key in self._ignore_keys: message.pop(key, None) if record.exc_info and record.exc_text is None: record.exc_text = self.formatException(record.exc_info) if record.exc_text: message[\u0026#34;exc_info\u0026#34;] = record.exc_text if record.stack_info: message[\u0026#34;stack_info\u0026#34;] = self.formatStack(record.stack_info) return json.dumps(message) The code is really simple, for each record you will get the dict from the record, and turn in to JSON with json.dumps().\nThere are only some conditions to add stack_info and exc_info if they are available, which should format exception info according to the record. You can easily modify that to fit your needs.\nAnd to use this formatter with the loggers:\nimport logging # import JSONFormatter logger = logging.getLogger(__name__) logger.setLevel(logging.DEBUG) handler = logging.StreamHandler() handler.setFormatter(JSONFormatter()) logger.addHandler(handler) logger.debug(\u0026#34;debug message\u0026#34;) which will output:\n{ \u0026#34;name\u0026#34;: \u0026#34;__main__\u0026#34;, \u0026#34;levelname\u0026#34;: \u0026#34;DEBUG\u0026#34;, \u0026#34;levelno\u0026#34;: 10, \u0026#34;pathname\u0026#34;: \u0026#34;main.py\u0026#34;, \u0026#34;filename\u0026#34;: \u0026#34;main.py\u0026#34;, \u0026#34;module\u0026#34;: \u0026#34;main\u0026#34;, \u0026#34;exc_info\u0026#34;: null, \u0026#34;exc_text\u0026#34;: null, \u0026#34;stack_info\u0026#34;: null, \u0026#34;lineno\u0026#34;: 38, \u0026#34;funcName\u0026#34;: \u0026#34;\u0026lt;module\u0026gt;\u0026#34;, \u0026#34;created\u0026#34;: 1663168021.864416, \u0026#34;msecs\u0026#34;: 864.4158840179443, \u0026#34;relativeCreated\u0026#34;: 1.2068748474121094, \u0026#34;thread\u0026#34;: 8673392128, \u0026#34;threadName\u0026#34;: \u0026#34;MainThread\u0026#34;, \u0026#34;processName\u0026#34;: \u0026#34;MainProcess\u0026#34;, \u0026#34;process\u0026#34;: 14747, \u0026#34;message\u0026#34;: \u0026#34;debug message\u0026#34;, } For list of all LogRecord attributes you can check the Python\u0026rsquo;s documentation.\nThat\u0026rsquo;s why I think for a code so simple, you are probably better off with implementing in your own code, rather than relying on a third-party package.\n","permalink":"https://aminalaee.github.io/posts/2022/python-json-logging/","summary":"Use-case It\u0026rsquo;s very common for projects to do JSON logging if you are working with third-party tools or open-source projects like Logstash to process your logs. These tools usually need more complex filtering on the structured data, so using JSON is preferred there.\nWe also wanted to integrate with a third-party tool at work and we needed to add the JSON formatting logs in our projects.\nI will not go into the details of why or why not you should decide JSON logging, as each approach will have it\u0026rsquo;s pros and cons.","title":"Python JSON Logging"}]