Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Cyclic imports and symbol dependencies #6

Open
yglukhov opened this issue Mar 11, 2016 · 17 comments
Open

[RFC] Cyclic imports and symbol dependencies #6

yglukhov opened this issue Mar 11, 2016 · 17 comments

Comments

@yglukhov
Copy link
Member

This is a feature request to allow cyclic imports. This allows to define mutually dependent types and procs (templates/macros) in different modules or in the same module regardless their definition order. Example:

# bar.nim
import foo

type Bar* = ref object
    f*: Foo

proc doSmthWithBar*(b: Bar) = discard

when isMainModule:
    let f = Foo.new()
    let b = Bar.new()
    f.doSmthWithFoo()
    f.b.doSmthWithBar()
    b.doSmthWithBar()
    b.f.doSmthWithFoo()
# foo.nim
import bar

type Foo* = ref object
    b*: Bar

proc doSmthWithFoo*(f: Foo) = discard

when isMainModule:
    let f = Foo.new()
    let b = Bar.new()
    f.doSmthWithFoo()
    f.b.doSmthWithBar()
    b.doSmthWithBar()
    b.f.doSmthWithFoo()

The symbol may be subjected to a cyclic lookup only if the following conditions are met:

  • The symbol is a top level symbol.
  • The symbol is defined by hand and is not a result of macro/template evaluation.
  • The type of the symbol is not dependent on another template/macro global evaluation.

Forum discussion: http://forum.nim-lang.org/t/2114

@endragor
Copy link

endragor commented May 17, 2016

  • The symbol is defined by hand and is not a result of macro/template evaluation.
  • The body of the symbol is not dependent on some template/macro evaluation.

Is it possible to ease these limitations somehow?

Why does body content matter here? I think large portion of Nim procedures are dependent on template evaluation - there are lots of them in stdlib and Nim promotes usage of templates to reduce boilerplate code.

@narimiran narimiran transferred this issue from nim-lang/Nim Jan 2, 2019
@timotheecour
Copy link
Member

timotheecour commented Feb 5, 2019

note: there is a (very) partial support for cyclic imports, see http://nim-lang.github.io/Nim/manual.html#modules

Recursive module dependencies are allowed, but slightly subtle

(see corresponding example + limitations)

@Zireael07
Copy link

I just ran into a similar limitation, where one type refers to another, sort of a child-parent relationship. Can this get fixed soonish? It's been open for several years...

@liquidev
Copy link

liquidev commented Aug 2, 2019

Any progress on this? It's really a pain when doing game dev, you have to split your modules in weird ways to achieve what you want.

@chr-1x
Copy link

chr-1x commented Aug 3, 2019

At the risk of making a post that basically boils down to "+1", this has been probably the biggest pain point for me with Nim so far.

To add a little more substance, here's something that's trivial to do in C/C++ and java-like languages (not to even mention scripting languages), but awkward to do in Nim:

widget.nim:

import processor

type
  Widget* = object
    processorOption: ProcessorOption

processor.nim:

import widget

type
  ProcessorOption* = enum
    ProcessFast
    ProcessSlow

proc process*(widget: Widget) =
  internalProcess(widget.processorOption)

Today, this yields the following error:

/cyclicnim/widget.nim(5, 22) Error: undeclared identifier: 'ProcessorOption'
This might be caused by a recursive module dependency:
/cyclicnim/processor.nim imports /cyclicnim/widget.nim
/cyclicnim/widget.nim imports /cyclicnim/processor.nim

What are the refactor options for fixing this error?

  • Move processorOption out of processor.nim into a third module: Awkward, because processorOption is clearly related to processing things, and users of processor will not appreciate having to add two imports to get the whole functionality of processor.
  • Move process(Widget) out of processor.nim into widget.nim: Awkward, because if we have the entry point for actual processing outside of processor.nim, what's it even for!? Besides, it calls internalProcess, which we clearly don't want to make public.
  • Move import widget in processor.nim below ProcessorOption declaration: Awkward, because now not all our imports are at the top of the file. Also, this doesn't generalize well to larger modules with more inter-dependencies.
  • Use include widget instead of import, with the {.experimental: codeReordering.} pragma: Awkward, because includes might not have the semantics we want (now we can access all the internal symbols!), widget doesn't have any visible reference to processor other than using its type, and now widget cannot be included on its own. Also produces this warning, despite compiling fine: Warning: Circular dependency detected. `codeReordering` pragma may not be able to reorder some nodes properely [User]
  • Just smash the two modules together, and reorder things yourself: Awkward, because these are primarily dealing with different things, even though they happen to refer to each other.

Not a solution:

  • Forward declare ProcessorOption in widget.nim: results in "implementation of 'ProcessorOption' expected". Making the enum value a ref, ptr, or seq is a non-option because they significantly change the semantics. This also applies if ProcessorOption was, say, an object.

How do other languages solve this problem?

  • In C/C++, there are a few solutions, none of them are particularly nice but they are well understood and frequently used: move the #include, forward-declare the enum (requires c++11 enum class so the compiler knows the size), or move the enum to a new header. The latter is less objectionable than creating a new nim module because it doesn't imply anything about visibility or linkage, and #includes are the default in C++. In any case, I really hope Nim can do better than C++ here. Also, I don't know enough about the new modules TS to comment on how it will handle this problem.
  • In C# (and maybe Java?), package-level namespacing and arbitrary-order declarations make this trivial. In C# you don't even have to explicitly import anything if they are declared in the same package. I would love if Nim had a package-style namespacing system like this.
  • In Python, duck typing more or less obviates the issue. If using type annotations, you can use from __future__ import annotations as of Python 3.7 to defer type resolution until later (putting the onus on tools to resolve the cycle, but they probably have more complicated symbol resolving logic anyway). Also, modules can be organized into packages using __init__.py files, which allows users to have a single import.
  • I'm less familiar with Javascript, but the ES6 modules appear to support such cyclic declarations without much effort on the part of the developer using explicit exports: https://stackoverflow.com/questions/46246383/cyclic-dependencies-in-javascript-modules-es6

@SolitudeSF
Copy link

Move processorOption out of processor.nim into a third module: Awkward, because processorOption is clearly related to processing things, and users of processor will not appreciate having to add two imports to get the whole functionality of processor.

you could just export the processorOption

@chr-1x
Copy link

chr-1x commented Aug 3, 2019

Move processorOption out of processor.nim into a third module: Awkward, because processorOption is clearly related to processing things, and users of processor will not appreciate having to add two imports to get the whole functionality of processor.

you could just export the processorOption

I was lazy with the export markers (*), I should have pasted my entire test file where I verified the problems described in the post. Unless you're referring to a feature I'm not aware of?

@SolitudeSF
Copy link

SolitudeSF commented Aug 3, 2019

i mean export processoroptionmodule from processor.nim so user doesnt have to import two modules

@krux02
Copy link
Contributor

krux02 commented Aug 3, 2019

First of all, I would really like to improve the usability of Nim for multi module projects. But I don't think we can do a prototype prepass like you describe here, and here is why:

The problem I see is, a function signature is not just what is visible at the first line. Nim also has an effect system. Many effects are inferred automatically. These inferred effects are attached automatically, but they are necessary to know, to compute the effects of other procedures. This also applies to forward declarations. A forward declaration should list all effects of that function. If it does not do that, the compiler won't be able to inject them in a later compilation stage.

@krux02
Copy link
Contributor

krux02 commented Aug 3, 2019

@chr-1x The way to deal with the current module system is to sort things by their dependency. This means if you have things that actually depend on each other, it is often a good idea to make it a single module. Put types that depend on each other in a single type section. But in your case, you don't have a cycling dependency, you can fix the cycle dependency by moving proc process from widget.nim to processor.nim.

@zah
Copy link
Member

zah commented Aug 4, 2019

@Araq keeps claiming that my original attempt to solve the cyclic dependencies problem through the noforward pragma is not compatible with the effects system, but I've never seen a compelling evidence for this. My own understanding to this day is that the algorithm described in the documentation can solve all problems:

nim-lang/Nim@1d29d24#diff-d86fb8f908ad34638ec054b961e99424R4651

The multi-module support with the noforward pragma has some challenges and the required refactoring in the compiler is quite substantial, but it's worth it IMO. It will bring some other benefits such as making nimsuggest significantly faster.

@al6x
Copy link

al6x commented Feb 20, 2021

Having same problem, hard to structure project and I forced to create artificial and unneeded modules like a.nim, b.nim and shared_ab.nim. Instead of just a.nim and b.nim

Some may argue that circular module dependencies are bad, like they breaks levelled architecture and have other problems. But here the case is different. Those problems are if you have long distance circular dependencies between different layers etc. And here we are talking about short distance circular dependencies, that are frequently is just a more convenient way to re-arrange code in large module into smaller chunks as separate modules.

@Araq
Copy link
Member

Araq commented Feb 22, 2021

We don't want to implement a solution that accentically prevents incremental compilation from working so the priority was put on IC. Now that IC is slowly beginning to work, we are looking into how to support recursive modules.

@johnnovak
Copy link

Now that IC is slowly beginning to work, we are looking into how to support recursive modules.

Great new, this has been a constant source of frustration for me (and I'm guessing for many others).

@avahe-kellenberger
Copy link

Has any progress been made on this front now that IC has been in the works for a while?
Curious about the progress, would love to have cyclic imports supported.

@arkanoid87
Copy link

I had to drop Nim on many projects due to this.
Without a proper solution, many software design patterns become a bottleneck as soon the project reaches a certain size.
Solution would be stick to procedural paradigm and strict idiomatic Nim, and say goodbye to flexibility and pattern mimicry when doing FFI

@zevv
Copy link
Contributor

zevv commented Jan 1, 2023

Stretch goal for 2023: #503

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests