How to test auto generated code in Python

Hi! In my free time I love to work with code generation, meta programming & etc. I have a bunch of libraries, one of them O!MyModels (https://github.com/xnuinside/omymodels), that generates different Python models from various inputs.

What is special in code generation that needs some special ‘testing’ way?

Normally the result of your code is some business logic or action. So you just test that your code works and produce the result as expected. But..

Code as a result of your code

In code generation your ‘product’ is a code.

First of all, you of course can test like “I expected this output”. This way you test that code generation method return text (code it is just a text, yes) that you are expecting to see.

sample of code generation output

For example (from one of the tests from O!MyModels library — https://github.com/xnuinside/omymodels/blob/main/tests/functional/generator/test_pydantic_models.py#L4):

This small test checks that if we send ddl input — we will get this string with python code as output.

Okay. But at the end of the day our code should be generated as a working code. Code that can be run as a python model without any issues.

Test that generated code really works

To be sure, what the result of our program is really working code we need to create a test that really executes code that we generated & the result of execution is worked as expected.

What do we need to do to execute this code in Python?

If you will just try to do ‘eval’ you will fail with indent issues and others. It is really tricky and not effective.

1. So first of all we need to save our code as a normal python module.
2. You need to import this new created module in runtime to get possible to call it anyhow during the test

To solve this 2 steps I created the pytest fixture (I usually use only pytest for tests):

https://github.com/xnuinside/omymodels/blob/main/tests/integration/pydantic/conftest.py#L14

Output of this fixture is a function that you can call to get a python module with code that you just generated.

You can use it in test this way (like here — https://github.com/xnuinside/omymodels/blob/main/tests/integration/pydantic/test_pydantic.py ):

This way we got a test that not just compares that result is a text with code that we expected, but we load this code and check that code really works.

Of course, this test can be improved — added checks, that properties really are set up etc. So it is just a sample.

I hope it was not boring & can help you if you work with code generation and need some inspiration!