Using Pydantic models instead of dictionaries increases maintainability in Python apps

The dictionary is a fundamental data structure in python and the uses of them occur often. We get parameters from a request to an endpoint, responses from APIs and internally have functions and methods to pass data to. The dictionary works in many of these cases and is easy to create. The ease of this creation comes at a cost of maintainability and clarity of the intentions and responsibilities of any data object.
In this post I outline the advantages of using pydantic to represent data internally and when integrating with external sources.

 

Objects have responsibilities

For dictionaries, you have to deal with the mental overhead of what the data is, where it came from and what it is used for.
Improved code readability

Pydantic models are explicitly defined, making it clear what data is expected and what types are required. This improves code readability, as the structure and constraints of the data are evident from the model definition.

Dataclasses and dictionaries can become unwieldy and hard to read, especially when dealing with complex data structures.

Stronger typing and validation

Pydantic models provide strong typing and validation out of the box. When you define a pydantic model, you specify the types of each attribute, and Pydantic will automatically validate the data at runtime. This ensures that your data conforms to the expected structure and types, reducing the likelihood of errors. It is also easier to add complex validation and keep it in the scope of the object

When using mypy for checking type safety and maintainability, the explicit types and nameing allow more coherent code and easier collaboration.

In contrast, dataclasses and dictionaries do not provide built-in validation. While you can add validation manually, it's easy to forget or make mistakes. Pydantic's automatic validation helps! I guess the case can be made that dataclasses and Basemodel are on par with the intention of the responsibilities through the class naming, the easier syntax in pydantic is easier to read

Example:

https://gist.github.com/jseller/65be3a2b30adc749496b6854dee097fc

Conclusion

For quality code that is easier to understand (in your own head and in PRs) try using the BaseModel of a dictionary and see the differences in the quality of your code. Dataclasses provided a big improvement in the developer experience but pydantic has taken this a step further.

Objects have responsibilities and behaviour: using a specific object that fulfils the responsibility of the object. Data objects are responsible for their attributes and validation of them. Full Stop! 

Use logical classes to use these data objects and maintain a cleaner separation of responsibilities. When unit testing the behaviour of using the logical object, the higher quality of the data objects used will show.

Portable code enables a smooth flow to production for the product

What is portability? The application or platform you are currently developing needs to work on your machine but also in the production environment where all the users are using it. The measure of Portability is the ability of the application to behave predictably in all environments. 

In this article, I talk about portable configuration and using code branching effectively with your team when developing software products. By adding some DORA metrics to your reporting, it's easy to quantify the effect portability has on your overall quality

why do I need to do this? 

The reliability of the software increases when it works in production without modification, so this effort starts when developing on your local machine. When developing an application there is going to be a need to build the app locally and deploy it to your production environment for the users to use the app. There could be testing, staging, and other environments along the way. Enabling some good practices for the configuration of the app makes developing and testing a lot easier and reduces the stress of deployment.

Keys to success:

• A new person should be able to pull, run tests, start app with just .local.env, and can hook into secrets in a vault when in dev, staging and production environments
• Keep environment config versions and use the environment to use a specific file.

how do I do this?

The application code shouldn’t change, the configuration of the code is what changes. This configuration is unique to each environment (local, dev, staging, production). Reduce the complexity of moving to production removes barriers and increases velocity

• Collaborative development with PRs
• The sooner the smallest architecture changes are established, the easier the validation in the build process. The maintainability of the codebase is fundamental to portable software
• Configuration of the System. 
• By deploying to the integration environment and testing the system, you can gain confidence that the system will work the same way in production
• use env files and keep secrets safe in cloud storage
• Configuration of the product
• Use Feature Flagging to limit access to incomplete functionality

• Align on the product level for releasing the user-facing feature. Release internally first so docs and support know what the customer is getting. 
• https://jseller.blogspot.com/2019/01/software-releases-are-lot-more-than-code.html

Branches of Code

How to integrate my work with the code from other developer teams? Use Trunk-based development for collaboration and velocity. This process allows integration of PRs and rapid deployment of the outcomes.

What is the trunk? it is the main branch and should match the deployed branch in the production environment. When changes are in the main branch, expect those changes to work immediately in staging and production environments.

• Ensures smaller changes
• features that aren’t ‘ready’, that don’t fulfill the full intention of the feature and don’t fully solve the user problem don’t need to be in a separate branch,
• Use feature flags to only allow users that have the alpha, beta or GA access, but remove them and tie them to the license source of truth for access and usage stats to understand the impact (and usefulness) to the user
• sometimes an alpha version is required to show progress and demoing the functionality through the alpha-beta-GA cycle 
• Prevents over-engineering through smaller changes
• Favours rebasing changes on the main branch instead of large merges. This keeps the history cleaner on the main branch and makes rollbacks much easier if needed. 

Feature Branches

• building a feature in a separate branch sounds like a good idea, but incurs a lot of maintenance effort when keeping the branch up to date
• frequent updating and merging
• merge conflicts

Release branches

• When a deployment to production is complete, the semver of the release is known. It can ease the transition to trunk-based development to create a Release branch to handle any hotfixes made to the production system between releaseses. This allows small changes in prod so users aren’t blocked by using the product while the large feature branches are merged and validated. 
• the same change is applied to the main branch with a test so the main branch has the changes integrated already for the next deploy

More on specific git commands here 

https://jseller.blogspot.com/2020/03/source-control-saves-time-and-complexity.html

Automation

Use automation to validate this portability and test the observability at the same time.

• Use a certified CI/CD for package and dependency validation

Use pipelines in bitbucket or github to automatically deploy your main branch to a development server, for larger production deployments there is Harness and other tools that can deploy to many nodes to reduce the complexity of setting up many production instances.

• Run tests appropriate to the environment

1. local tests and mock data for local development
2. Integration tests and synthetic data for staging
3. Production smoke tests to validate

Use an automated test to add a quality step to your deployment scripts. The deployment is finished when it is validated, not just deployedValidate your observability when validating the deployment. This is essential to certify the system is up to date for deprecation or security fixes

Metrics to measure?

Good portability helps velocity and collaboration, but how does it do that? Use DORA metrics to understand how the portability is affecting deployments and rework

• Change Failure Rate
• Deployment frequency

https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance

Conclusion

No matter how careful you are about the portability of the app; things are going to happen when many people are working on the application. 

Adding quality through Portable patterns like code branching and automation will go a long way to ironing out the issues before deploying to production

Also see:

https://jseller.blogspot.com/2019/01/feature-definition-with-quality-model.html

https://jseller.blogspot.com/2020/03/packaging-code-for-development-and.html

https://dora.dev/capabilities/trunk-based-development/

How individual contributors add value to technology teams when not managing people

Building software products is a team sport, and there are many positions to play on any team. In software teams, technology leadership positions have evolved over the years from primarily people management to a combination of personnel and technology management.  It’s difficult for one person to fulfill these distinct responsibilities successfully; as a result, successful teams have split this role into two:  Engineering management is people management, and Staff development is focused on technology production. 

The success of the team is dependent on the people involved, and the understanding of the goals they need to achieve. The Engineering manager enables an understanding of these goals, enabling teammates to play their positions more effectively. This develops the person. 

The individual contributor isn’t the coach; they are more like the team captain and they show by example how to fulfil the responsibilities of the position. This brings other developers up to a higher level of quality development. The Staff developer moves the technology forward. While the specifics can vary from team to team, at their core, they must fulfill the fundamentals of: 

• Technical excellence, by example
• Predictability of delivery for stakeholders 
• Ensuring Architecture quality through collaboration on design and review 

Technical Excellence, by example

A staff developer must show technical excellence by example and how to move development forward with quality and velocity. This effort is not just building a feature, but engineering the feature so that it has the quality needed for the product to be successful in the market. 

Starting with the code and showing guidance and practices for maintainability and portability with pull requests that follow a declared set of quality attributes that guide quality code. What is quality code? Code that follows common conventions that the group has decided on. This is formatting, linting and type safety so the code in the product is something everyone can recognize. Remember that the code isn’t the developer’s code, it is company property and must be maintained responsibly so that any developer can work on it. Working through pull requests by making sure the collaborators can review and push the code to production in a common way. There will be unknowns to figure out, but it’s necessary sometimes to run into the dark room and light it up together.

https://jseller.blogspot.com/2020/03/packaging-code-for-development-and.html

https://jseller.blogspot.com/2023/06/maintainable-and-portable-codebases.html

Staff developer Improves predictability by reducing scope. 

Understanding the context to help build features for the user that are valuable. To move the product forward, understanding the impact of the decisions helps the predictability of the process and communication during design implementation

The smaller the better and output an implementation Epic to understand the definition of done for this enhancement. Work that implementation to production.  Use feature flags to enable this functionality to the user when releasing to the production environment

https://jseller.blogspot.com/2019/01/software-releases-are-lot-more-than-code.html

Ensures architecture quality and collaboration on the definition of that quality

Operationally the Staff developer ensures a high level of adherence to the quality attributes of the system. The functionality, does it have some understanding of the problem we are solving for the user? 

Collaborate on functionality, observability, and the tools and processes that enable easy maintenance of the product in production. Adhering to known standards establishes good practices and processes. 

https://jseller.blogspot.com/2019/01/feature-definition-with-quality-model.html

Does the solution have a coherent definition of the functionality and acceptance criteria? Staff developers know they can’t build the whole thing in one change, but able to break down the solution into smaller chunks and gain agreement on the smallest definition of done that shows the value

• https://jseller.blogspot.com/2023/06/effective-software-architecture-meetings.html

Evolving the architecture together

To support any functionality there will be changes to the architecture. Develop a strong culture of design through a standardized process of RFC (or ADR) review in the team. 

• system requirements map to unit tests so the understanding of the scope is clear. 
• build tests from the acceptance criteria for integration testing in the development and staging environment
• build sanity/smoke tests to validate this functionality in the production environment
• build observability, performance, security, and portability from these small changes, 
• Working with living specifications enables async collaboration. Keep these documents concise and to the point. 
• Good design ‘fits in’ to the system, use diagrams for containers, and components. that are new or changes
• https://jseller.blogspot.com/2023/06/consistent-software-design-enables.html
• https://jseller.blogspot.com/2020/02/communicating-software-architecture.html

Wrap-up

Staff development brings an awareness of the product and company’s goals to technical development. This ensures the teammates can align on the architecture and collaborate at scale to deliver a quality product. This direction, and most importantly, the collaboration with the team members, ensures that the product produced can fulfill the goals of the company’s place in the market.