Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter between mirrors #608

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

mswart
Copy link

@mswart mswart commented Aug 2, 2017

Fixes #357

Description of the Change

Distributions contain a huge set of packages, mirroring them requires a large amount of storage and bandwidth. To mitigate this issues aptly provides an filter feature to reduce the set of mirrored packages. Especially it supports to limit the downloaded packages to a set of packages and their needed dependencies as well.

As this filter is self-contained within each mirror, it has two limitations:

  1. its difficult to partial mirror a security or updates mirror to match a filtered main repository. Update and security repositories contain (newer) packages for a (small) subset of the whole distribution to fix bugs/security issues. As the repository contains not all packages, filtering breaks due to cuts within the dependency tree.

  2. integration of packages from other sources. It is not possible to filter for all dependencies of a package if this package itself is not part of the mirror repository but instead published within another (local) repository.

In both cases all dependencies between repositories/mirrors have to be included manually in the mirror filter.

This commit introduces -deps-from-mirrors and -deps-from-repos options for mirrors. They are comma-separated lists referencing mirror/repository names.

If -filter-with-deps is activated all dependencies of packages from the references mirrors/repositories that are part of the current mirror are included in addition to the filter result itself. This resolves 2.

If in addition -dep-follow-all-variants is activated, all packages that are included in the referenced mirrors/repositories and present in the current mirror (matched by package name only) are selected to download, too. This resolves 1.

Limitations / Questions

  • I am not confident about the option names, but I lack ideas for shorter, more to the point ones.
  • The implemented approach differs from the proposals in -filter-with-deps across mirrors #357 - each mirror is updated on its own. If both mirrors have (many) packages with dependencies only in other mirrors, it might take some update rounds (update mirror 1, update mirror 2, update mirror 1 ...) to pick up all needed dependencies. I observed a similar behavior for experiments with xenial-mirrors and its security/update repositories. It took 2-3 updates to reach a stable filter point. I am currently unsure about the cause, I wound investigate further if the feature / fundamental implementation design is approached.
  • I am unsure how to best implement functional tests.

Checklist

  • unit-test added (if change is algorithm)
  • functional test added/updated (if change is functional)
  • man page updated (if applicable)
  • bash completion updated (if applicable)
  • documentation updated
  • author name in AUTHORS

@smira smira added the 1.2.0 label Aug 17, 2017
@smira
Copy link
Contributor

smira commented Aug 17, 2017

Thanks, looks interesting!

@mswart
Copy link
Author

mswart commented Aug 18, 2017

Nice. I have rebased the PR to fix the merge conflicts.
I am happy to make further adjustments or addition to your feedback.

@smira smira added 1.3.0 and removed 1.2.0 labels Dec 5, 2017
@belm0
Copy link

belm0 commented Mar 13, 2018

I tried the patch with my use case and it worked well-- looking forward to the feature.

It's been a while since the last rebase, would you run again?

@hsitter hsitter removed the 1.3.0 label Apr 30, 2018
@lbolla lbolla closed this Jan 28, 2022
@erikoest
Copy link

Is this patch available for aptly version 1.5.0? I tried to merge the mswart:filter-between-mirrors branch over aptly-dev:v1.5.0 and got a few conflicts, mainly in the api code.

@neolynx
Copy link
Member

neolynx commented Oct 10, 2024

Hi !

It looks like this was never merge, just closed probably..

should we reopen this ?

@erikoest
Copy link

It's up to the aptly-developers, I guess. And I think a new merge req is needed since this one won't fit with newer versions of aptly. I'm using the filter-between-mirrors functionaliy actively, so I'll need it in later versions. But first, I can ask mswart if he would like to maintain his branch and make it compatible with v1.5.

@neolynx
Copy link
Member

neolynx commented Oct 10, 2024

that would be great... let's have a look at how good it rebases...

@neolynx neolynx reopened this Oct 10, 2024
@neolynx neolynx self-assigned this Oct 10, 2024
@neolynx
Copy link
Member

neolynx commented Oct 10, 2024

rebased...

@neolynx neolynx added the fix tests Tests are failing label Oct 10, 2024
@neolynx
Copy link
Member

neolynx commented Oct 10, 2024

does not seem to build anymore on master

@neolynx
Copy link
Member

neolynx commented Oct 10, 2024

@erikoest
I read about some issue here: #357

were they fixed, could they still be an issue ?

@erikoest
Copy link

I'm not familiar with issue #357. I don't think we've seen this problem in our systems. Maybe you should consult @mswart who wrote the patch.

@neolynx
Copy link
Member

neolynx commented Oct 11, 2024

I managed to fix the build and the tests :)

coverage is low, so we need more tests: 38.52% of diff hit (target 74.84%)

@mswart I know it has been years, would you still be interested in taking this further ?

@neolynx neolynx added increase coverage The PR lacks test coverage help wanted ! Please help getting this PR merged :-) and removed fix tests Tests are failing labels Oct 11, 2024
@neolynx neolynx removed the wishlist label Oct 22, 2024
mswart and others added 3 commits October 22, 2024 22:09
Distributions contain a hugh set of packages, mirroring them
requires a large amount of storage and bandwidth. To mitigate this
issues aptly provides an filter feature to reduce the set of
mirrored packages. Especially it supports to limit the downloaded
packages to a set of packages and their needed dependencies as well.

As this filter is self-contained within each mirror, it has
two limitations:

1. its difficult to partial mirror a security or updates mirror to
   match a filtered main repository. Update and security repositories
   contain (newer) packages for a (small) subset of the whole
   distribution to fix bugs/security issues. As the repository
   contains not all packages, filtering breaks due to cuts within
   the dependency tree.

2. integration of packages from other sources
   It is not possible to filter for all dependencies of a package
   if this package itself is not part of the mirror repository
   but instead published within another (local) repository.

In both cases all dependencies between repositories/mirrors have to
be included manually in the mirror filter.

This commit introduces `-deps-from-mirrors` and `-deps-from-repos`
options for mirrors. They are comma-separated lists referencing
mirror/repository names.

If `-filter-with-deps` is activated all dependencies of packages
from the references mirrors/repositories that are part of the
current mirror are included in addition to the filter result
itself. This resolves 2.

If in addition `-dep-follow-all-variants` is activated, all
packages that are included in the refereneced mirrors/repositories
and present in the current mirror (matched by package name only)
are selected to download, too. This resolves 1.
@neolynx
Copy link
Member

neolynx commented Oct 22, 2024

rebased on master...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted ! Please help getting this PR merged :-) increase coverage The PR lacks test coverage
Projects
None yet
Development

Successfully merging this pull request may close these issues.

-filter-with-deps across mirrors
7 participants