Mojo/Python project folder structure

[EzRyder]

I originally posted this on Discord (link), but @DarinSimmons felt it would make a good topic for this forum.

I’m looking for guidance on folder organization for a significant Mojo/Python project. I’m importing standard Python objects/libraries and custom Python modules throughout my codebase, and all the tests are written in Mojo. I use Python.add_to_path extensively so Python imports can find the custom modules. For testing, I added a Symlink in the tests folder which points to the code folder so the tests can find the source files.

Referring to the folder structure outlined below:

1. Is this a reasonable folder structure? Any suggestions?
2. Any alternatives to Python.add_to_path so Mojo can find the custom Python modules for import?
3. Any alternatives to using a Simlink in the folder tests so the source files can be found by tests?

TIA for any suggestions/guidance!

top
|-code
|   |-mojo_shared
|   |   |__init__.mojo
|   |   | # <.mojo files>
|   |-py
|   |   |-gui
|   |   |   |__init__.py
|   |   |   |gui_main.py
|   |   |   | # <.py files>
|   |   |-common
|   |   |   |__init__.py
|   |   |   | # <.py files>
|   |   |-oauth
|   |   |   |__init__.py
|   |   |   | # <.py files>
|   |   |__init__.py
|   | main.mojo     # <-- Execution starts here.
|   | # <.mojo files>
|-icons
|   | # <icon files>
|-logs
|   | # <.log files>
|-tests
|   -code           # <-- Symlink
|   | # <test_*.mojo files>
|.gitignore
|top.code-workspace
|magic.lock
|mojoproject.toml
|mypi.ini
|README.md
|test.sh

[jack.clayton]

Hi @EzRyder the Mojo directory structure works the same as Python, you no longer have to use add_to_path as it adds the directory of the source file/binary to sys.path for the python interpreter. There is a blog that explains it: Modular: Using Mojo🔥 with Python🐍 if you scroll down to the title What is a package?

Python packaging file structure is not very intuitive, if you send me a link to your github repo I can add a PR to fix the imports/file structure and remove add_to_path

[EzRyder]

Hello @jack.clayton! I just created a repro on Github that we can use to resolve my Python add_to_path issue. I invited you to join the project.
In the file as_paths.mojo are functions that add several paths to sys.path using Python add_to_path. Specifically fn add_test_paths and fn add_main_paths. You’ll see in the test files where add_test_paths is called. There’s a task in the mojoproject.toml file to run all tests using magic run tests in a terminal. This actually calls a small shell script, test.sh.
Important: Currently, you’ll need to place a symlink into the tests folder pointing to the AsanaDL folder for the tests to run/succeed.
Run the file main.mojo to start the app. This opens the Python GUI. I’ve stripped out the application functionality for this repro, so nothing happens when clicking around on the GUI.
I think that’s it as a start. Let me know if you have any questions. We can talk or have a video chat with screen sharing if that will help.
Thanks!

[jack.clayton]

Thanks for the repro, I have a local branch with most of the paths fixed up, but getting this error when trying to run main.mojo:

 No such file or directory: '/Users/jack/src/repro_EzAsanaDL/logs/logAgent.log'

Getting this error without any changes as well, Is there a way to create the log file?

[EzRyder]

Edit: I just added automatic creation of the logs folder if it does not exist. I modified file my_logger.py and synced the change to Github.

My first guess is that you have to manually create the logs folder.
~/EzAsanaDL/logs

[jack.clayton]

Thanks, the tests are passing on https://github.com/johnsoez4/repro_EzAsanaDL/pull/1/files

I can’t run the GUI as I’m on a remote linux machine, the PR isn’t for merging but you can checkout that branch and get an idea of how to structure the project in the way that mojo/python expects without using add_to_path. Also added a magic command to run mojo test without using the shell script, and another to run mojo main.mojo

Moved the files around to simplify things a bit as I fixed the imports, but you can move things back one at time and fix the imports.

Hopefully this helps, let me know if you have questions.

[EzRyder]

I’ve made the changes shown in your PR for the repro, with minor changes to the folder names. The tests run successfully, but when I try to run main.mojo it fails on the Python.import_module (see below). I’ve played around with it but can’t find a fix.
I’ve committed all changes to Github if you want to sync and take a stab at it.

# It fails on this line in main.mojo.
var py_gui_main = Python.import_module("AsanaDL.py_agents.gui_main")
# Error: Unhandled exception caught during execution: No module named 'AsanaDL'

[jack.clayton]

There are a couple of limitations I detailed in the PR description here: https://github.com/johnsoez4/repro_EzAsanaDL/pull/2

[EzRyder]

I’ve updated the repro per feedback in PR#2, and the GUI now starts then closes with the error Unhandled exception caught during execution: No module named 'py_shared'. I confirmed that this is in fn agent of module py_utils.mojo on the statement Python.import_module("py_shared.py_agent"). I have not touched the folders containing the Python modules.
Referring to your comment in PR#2 - can Mojo code NOT find custom Python modules to import if they are in a subfolder to the Mojo code? This cannot be correct, right?

[EzRyder]

Figured it out…the answer was already in main.mojo. Import needs to look list this:
Python.import_module("py_shared.py_agent") # KO - No module named ‘py_shared’
Python.import_module("AsanaDL.py_shared.py_agent") # OK - Twas missing AsanaDL.

[EzRyder]

Here’s the folder structure that works for the repro project. I will be transferring the lessons learned into my real project.
Imports

Python.import_module("code.py_gui.module1")
Python.import_module("code.py_oauth.module2")
Python.import_module("code.py_shared.module3")

Folder Structure

top
|-code
|   |-py_gui
|   |   |gui_main.py
|   |   | # <.py files>
|   |-py_oauth
|   |   | # <.py files>
|   |-py_shared
|   |   | # <.py files>
|   |__init__.mojo
|   | # <.mojo files>
|-icons
|   | # <icon files>
|-logs
|   | # <.log files>
|-tests
|   -code  # <-- Symlink [NOT required, but useful for running single test files]
|   | # <test_*.mojo files>
|.gitignore
|top.code-workspace
|magic.lock
|main.mojo     # <-- Execution starts here.
|mojoproject.toml
|mypi.ini
|README.md
|test.sh