Python's import system would be my biggest complaint about the language. I've used Python for more than ten years, but there are so many pitfalls with regards to imports that I still have to look things up on a regular basis. What makes things even worse is that packages sometimes even meddle with how imports work. (See e.g. Tensorflow – I recently had to debug the import magic they're doing. It was a nightmare.)
Don't get me wrong, I LOVE Python! It's my go-to language for both small and big tasks. But sometimes I do wish Python had a function like PHP's require()… (Obviously for package-local imports, not for importing other packages.)
Lua uses a "require()" function instead of an import statement for loading modules. I think that this is a crucial difference from a lot of programming languages that is vastly understated. What this means is that you can completely replace "require()" with your own function that has complete control over how the module importing system works. Because of this it is possible to build your own runtime code hotloader entirely in Lua without needing to modify the source of the compiler, and thus Lua is the only "mainstream" programming language listed in the list of notable live coding environments on Wikipedia[0]. (I'm surprised that Clojure isn't listed there, though.)
The only way this can be accomplished is by designing the import system from the start to use a function instead of a statement. It's too late for Python to redo things and change its import system, unfortunately. It really is the one language feature you have to get right the first time or you'll miss out on those kinds of enhancements forever.
Sorry, I think you're right. I think the actual problem I was trying to get at is that Python makes it easy to both import from a module and bind the imported functions/values to local names at the same time. This means the values get bound privately in the module, instead of always being late bound by accessing the module as a dictionary. And because the code in libraries uses this way if importing things, you can't get at the old names if you try to hotload a module.
The docs for Python's importlib say as much:
> If a module imports objects from another module using from … import …, calling reload() for the other module does not redefine the objects imported from it — one way around this is to re-execute the from statement, another is to use import and qualified names (module.name) instead.
But Lua isn't actually different here since you can still bind the members of an imported module to local names manually, it's just that there is no "from" keyword that does this for you. My personal theory is because of the lack of something like "import * from" in Lua, it became more common for library authors to use late binding and access the returned module by indexing into it. However this is just my personal theory.
Also, now that I think about it you still have to make adjustments to your code in order to have hotloading be feasible, even in Lua. I deliberately chose to use late binding in the code I wrote with "qualified names" (really just table indexing). And all the third party dependencies are versioned in the project's repository, and don't have dependencies on other external modules, so there was no need to worry about a dependency breaking hot reloading because it uses imports differently. I think this might be a cultural thing. A lot of Lua libraries are written in a single-file, no-dependency manner which makes them much easier to integrate in the ways I want. Python seems to have a much larger ecosystem with many external dependencies depending on others.
So yes, I think it makes sense that Python could have module reloading, but the problem is that the language encourages the use of local binding from imports which is incompatible with it, and there is already a lot of code that imports things like that, so it's not as practical if you're using libraries from pip or use "import <...> from" anywhere in your codebase.
I think you have problems if you just write local foo = require ("foo") in two different files and you want to replace the module object in both of them at once.
Edit: the below is probably wrong about what you're doing. It sounds like your require-replacement merges the new return value of a module into the old one, so code that accesses stuff through the module's top level table will get the right result.
Wrong stuff: Maybe the hot reloading system you use relies on modules mutating the module object rather than making a new one. This is pretty unusual among Lua modules though. I usually see people unconditionally make a table and return it.
Yes, the table merging is what mine actually does. It clears the old table reference and inserts the contents of the new module into it. It seems to work well enough for my use-case.
Unless something has changed or there's a detail involving builtins I'm not aware of, I don't think this would work, for the same reason as the most common pitfall when mocking functions in python: You have to replace it where it's used, not where it's defined. To do that on a module, you have to first import that module - and then all the imports at the top of it would've already run.
> To do that on a module, you have to first import that module - and then all the imports at the top of it would've already run.
Why not just replace the `__import__` function at program startup, when you haven't imported anything? Monkey patching `__import__` at any other time seems like a bad idea.
I recently wrote a hot reloader for Lua[0]. It's actually a port of something I had done a decade ago in Python! Last time Python hot reloading came up on HN, I asked the author about this approach, and the only problem seemed to be that it was not so great if one was accessing the values one wanted replaced from multiple threads. Lua avoids this problem only by not having a multithreaded interpreter.
`require` is even better than you've indicated: it simply calls the contents of `package.loaders`, in order, on any new string it's given.
So in many instances, there's no need to replace the require builtin: one may simply add a new function to `package.loader`, anywhere in that order that's useful.
I've done this, and draw most (basically all) modules out of a SQLite database. It works like a charm.
If one has need to invalidate the cache for a given module, as comes up in live coding environments, simply set the require string in `package.loaded` to `nil`. Small caveat: the `package.loaded` table is 'special', in that the C which `require` is written in hard-codes a reference to it: one may not simply replace `package.loaded` in the global namespace and expect the new one to be referenced.
> `require` is even better than you've indicated: it simply calls the contents of `package.loaders`, in order, on any new string it's given.
Python’s import goes through the configured finders (sys.meta_path) and invokes them in-order with the name of the package being imported until one of them returns an import spec. Sound ‘bout the same to me?
I’ve written loaders which created “virtual” modules on-demand in the past.
The discussion is about python and the comment you’re replying to specifically (and largely incorrectly) trying to contrast lua’s require with python’s import?
I recently spent an entire day trying to import AllenNLP and its dependencies. I gave up after trying multiple interpreters, virtual environments, and operating systems.
That was probably the first things that attracted me to Docker.
If I started a new job or new project, it was the same story every time. “Do these five things and then you should be able to run the code” after half a day of back and forth I’d have a list of ten things, and the next new person would discover it is actually 12.
Docker doesn’t just change your experience, it forces the author to actually capture most of their prerequisites. To the point where even if you don’t use docker, you benefit from a project having at least tried to use it.
Frankly I was just trying to get a pre-trained ELMO model deployed via Cortex. As best I could tell the problem boiled down to a failure related to installing tensorflow with gpu disabled. I am a NN noob but still, it just shouldn’t be this hard.
Unfortunately that, whilst seeming simple-enough on the surface is one of those things that cuts across a whole number of layers, which makes it quite involved to solve any issues that arise.
Inside the source code files, they then use a decorator (@tf_export) to define where in the package/module hierarchy a function/class should get placed, see e.g.
For instance, the write() function in that module actually gets exposed as `tf.summary.write()` even though it's contained in the file tensorflow/python/ops/summary_ops_v2.py.
Needless to say, this even brings auto-completion of an IDE like IntelliJ to its knees at times.
To make things even worse, things like Tensorboard also make use of global state everywhere, see e.g. the function _should_record_summaries_internal() in the above file. But now the question is: Which module's global state does the function actually access if it gets "moved" to a different module by TF? At the end of the day, I gave up hunting down that bug (_should_record_summaries_internal() always returned False, no matter what) and resorted to mocking the function out whenever I used Tensorboard. To do that (i.e. to find its parent module in the first place), I in turn used the `inspect` package.
I ran into that issue a few weeks ago. I had a small project that was supposed to migrate rows from a MySQL server into a SQL Server instance. For some reason that I was too exhausted to figure out, the program would only work if I imported the mysql connector before the pydobc stuff. That was very frustrating to say the least.
I've seen this complaint for almost every language, and the few languages i don't see this complaint for, people end up complaining about something else. It's too late to change how imports work and it's not especially complicated so no use complaining.
I pretty much never have problems with imports in Ruby. It's not like it's an unsolvable problem. Nothing would ever get better if we just took the "no use complaining" approach.
Well, to be fair, Tensorflow is a Google library - pretty much any Google-made library will have five levels of indirection before it does anything, regardless of language. It's in their DNA.
> I ask because from what I understand from Google's monorepo, it should be easy for engineers to take direct dependencies on whatever they want.
Not correct. The dependency system supports visibility restrictions - you can take direct dependencies to anything only if you're in your own experimental tree. Otherwise package authors can control access.
I worked at Google for several years until 2015 - one of the running inside jokes was that increasing a border by one pixel is a full-quarter project with a PM, a tech lead, and multiple engineers.
It's not like most people would want more complexity (at least I hope not), but somehow it kept happening, and I feel at least some of it is cultural.
At the risk of over generalizing, I think 2 things are going on:
1) They hire a lot of smart but inexperienced people. I've noticed that those kinds of programmers often write clever and overengineered things, which means overly complex.
2) They get promoted for creating complicated things. They don't get promoted for writing simple things.
Don't get me wrong, I LOVE Python! It's my go-to language for both small and big tasks. But sometimes I do wish Python had a function like PHP's require()… (Obviously for package-local imports, not for importing other packages.)