unnecessary library dependency

[ssulca]

If you would like to use muttlib only for ELT using functions like

• dbconn
• muttlib.utils

So for these case scikit-learn is not necessary

[lahgzz]

If you would like to use muttlib only for ELT using functions like

• dbconn
• muttlib.utils

So for these case scikit-learn is not necessary

Hi @ssulca 👋, and thanks for submitting this issue!

Yes, you're totally right! We're aware of some problems in the dependency management department at muttlib and this is good example.

I take this to tell you that we've been working on implementing poetry as our main packaging and dependency management tool. We believe this is going to help boosting muttlib's dependency management capabilities and hope to mitigate this sort of issues.

Feel welcome to keep bringing ideas and/or contributions to muttlib. If you want to know how, you can follow our CONTRIBUTING guidelines for more info.

Thanks again!

[leo-mansini]

I'd set sqlalchemy, scipy and other libraries as optional, or maybe we can take advantage of poetry dependency groups, wdyt @aoelvp94 ?

[aoelvp94]

@leo-mansini is sqlalchemy core for our adapters? is yes, I think sqlalchemy should be a core dependency. WDYT? Do you have plans to have a replacement like prisma https://github.com/RobertCraigie/prisma-client-py instead of having sqlalchemy as ORM?

[leo-mansini]

I agree it's core for adapters, but I guess it's not certain to me whether muttlib is a database conector library or a general helpers/utils lib.

If it is the latter, why would someone want to have sqlalchemy if, for example, they are only using path_or_string from utils? There was a project where this was the case.

Regardless, there are libraries not needed for most connectors, such as the snowflake lib.

[aoelvp94]

Agreed, IMHO we should have muttlib (maybe renamed as muttlib_dbconn) for databases/data sources stuff and another brother library for airflow stuff, another one for mlflow stuff, another one for python utils, and so on. WDYT? Should we put a poll on Slack to debate on this?

[leo-mansini]

Agree on asking for opinions. A kind of split (in repos or modules) is def needed IMO.