Separate repositories for each egg

[benoitbryon]

Create one repository for each egg packaged in current src/ folder, i.e:

• thumbnailer.core => could be a rename and refactor of this repository to host only thumbnailer.core package.
• thumbnailer.engines.images => new repository required
• thumbnailer.engines.documents => new repository required

[benoitbryon]

Discussion began in #13...

benoitbryon said:

It seems that pip looks for a setup.py file at repository root. But currently we can't do that because we host several eggs. I guess the best solution would be to create one repository for each egg, i.e. 3 repositories, one per directory in src/ folder.

Natim replied:

Maybe one setup is en enough to install the three packages. I don't think it is a good idea to create three repository.
If it is pluggable it will allow other project to create other engines. But we can provide some in this repository.

[benoitbryon]

From my point of view, creating one repository per egg is a good choice to allow asynchronous maintenance of each package:

• separated bugtracker. As an example, issues in images engine don't block thumbnailer.core development.
• separated documentation, i.e. install procedure (system dependencies)
• separated releases, i.e. versions. As an example, development branches for releases of thumbnailer.engines.images should not affect thumbnailer.core.
• separated tests. As an example, we don't want a regression of images engine to break thumbnailer.core tests, and vice-versa.
• optionally, separated maintainers/contributors.
• maybe, one day, we will deprecate one part. As an example if someone creates a better images engine, why would we keep on distributing thumbnailer.engines.images with thumbnailer.core? If most people don't need all engines, then they don't want to pull all the code.

[benoitbryon]

Additional note: if we separate repositories, we should have common conventions. So that contribs look like core.
As examples:

• documentation template and conventions
• tutorial about creation of a new engine

[benoitbryon]

For users, what does separated packages mean?

They will be able to do:

pip install git+https://github.com/Natim/thumbnailer.core@master#egg=thumbnailer.core
pip install git+https://github.com/Natim/thumbnailer.engines.documents@master#egg=thumbnailer.engines.documents
pip install git+https://github.com/Natim/thumbnailer.engines.images@master#egg=thumbnailer.engines.images

Or, if we publish these packages on Pypi:

pip install thumbnailer.engines.images thumbnailer.engines.documents

(where thumbnailer.core will be automatically downloaded as a dependency)

[benoitbryon]

What are alternatives to separated repositories?

1. One setup.py to distribute three packages => I think it is impossible, but maybe I'm wrong.
2. Wait for pip's "subdirectory" option. See https://groups.google.com/group/python-virtualenv/browse_thread/thread/4296d43d608f0ace/b93e48acb255332c?show_docid=b93e48acb255332c => I can' wait, I want to use thumbnailer now :)
3. Restore thumbnailer package with core and engines subdirectories (i.e. implement only Create a Python package for thumbnailer (setup.py) #6) then try to create a contrib that plugs in thumbnailer.engines namespace => not sure it works, but we can give it a try.
4. Forget pip and have a look at recipes for buildout => I guess we need both pip and buildout support. So, not a good idea.
5. @Natim: any other idea?

[Natim]

My solution is to separate thumbnailer with the architecture you did and make one setup.py that will install the basic (so thumbnail.core, thumbnail.engines.*) but allow third party to create new engines and be able to plug them.

For me it doesn't matter that the thumbnailer package provide some basic engines in its package.

[benoitbryon]

It sounds like solution n°3, isn't it?

1. Restore thumbnailer package with core and engines subdirectories (i.e. implement only Create a Python package for thumbnailer (setup.py) #6) then try to create a contrib that plugs in thumbnailer.engines namespace => not sure it works, but we can give it a try.

Let's create a branch to implement #6, then create another repository (to implement a fake engine or #1 as examples) that tries to plug in thumbnailer.engines namespace.
So we will see if it works.
If it doesn't work we could:

• create distinct repositories
• or use some thumbnailercontrib namespace for contribs (i.e. something not packaged in thumbnailer base package)

[benoitbryon]

<troll>

For me it doesn't matter that the thumbnailer package provide some basic engines in its package.

That's a Django-user point of view :D
Batteries included, yes.
But enhancements of included apps are released with the full package, i.e. later, even if they have been proven stable for months (or years).

</troll>

In fact, I feel images and documents engines are not so "basic". One requires PIL and the other requires pydocsplit. Both have some system requirements.
It means that the thumbnailer's setup.py have to mention "pydocsplit" and "PIL"... which are not really required if the user chooses to use some pure Python engine.
From my point of view, a "basic" engine would be a pure Python engine, with almost no additional dependency. It would be a kind of demo or fake engine (useful for tests!).

[benoitbryon]

Created a 6-one-python-package branch to review the "batteries included" way.

As told in #6:

I personally dislike the presence of PIL and docutils as thumbnailer dependencies (see thumbnailer's setup.py): if we just want to install custom engines like "thumbnailer.engines.same", we don't want dependencies of thumbnailer.engines.images and thumbnailer.engines.documents (i.e. PIL, docsplit and their respective dependencies).

From my point of view, if thumbnailer.core is pure Python I can develop it directly on my machine, within some isolated virtualenv. Whereas I create a virtual machine if I need to install docsplit and PIL dependencies. Because I can't manage clean isolation of non Python package on my system.