remove the need for svn:externals (and merge the icetray repository back into the main one)

[Claudio Kopper]

We discussed the possibility of removing the need for svn:externals in the usual meta-projects at the software session in Copenhagen (they might still make sense for more specialized uses and/or meta-projects). The idea is to move all relevant project trunks directly into "combo" and start releasing the whole meta-projects. This makes branching/merging/... work in a sane way.

The process requires re-integrating the "icetray" repository into the main repository. A way that makes this work without breaking old meta-projects using externals is to clone our main meta-project and set the existing ones to read-only. The new meta project would exist at a new URL, something like ​http://code.icecube.wisc.edu/icecube instead of ​http://code.icecube.wisc.edu/svn and ​http://code.icecube.wisc.edu/icetray .

Attached is a working list of commands that would need to be run to make this work. I tested them on a cobalt yesterday. You can find a repository created this way on cobalt02 in /scratch/ckopper/ (the repository URL is ​file:///scratch/ckopper/restored_dump (on cobalt02, not on a shared filesystem because Lustre is very, very slow when storing these).

With SVN sparse checkouts users can retrieve partial copies of the meta-project. An example is shown at the end of the attached script.

Please send comments on what you think about this. I think it would make the release process a lot easier and allow for things like svn diffs of whole meta-projects.

[Alex Olivas]

svn diffs of meta-projects are always going to be gnarly. I generated one in an attempt to debug The Leif Problem and after 40 pages and 3 hours at a cafe, I got no where. I don't remember how I did it, but I don't remember it being that tough to generate.

The release process isn't going to be made that much easier with this. Where most of the delay comes from is the cat herding component. Asking project maintainers "Is this really ready for release?"...waiting for the reply or last minute tweaks, then testing and finding something's actually broken that wasn't covered by any test. The series of 'svn cp's take all of about 10 min...maybe an hour for IceRec?, but is a negligible factor in the weeks to months it sometimes take to get releases together.

In the end, isn't this going to encourage (or at least in no way discourage) a tighter coupling between projects that were intended to be modular and independent?

[David Schultz]

Tickets that would benefit from easy repository-wide branching: #985, #1538, #1002, #1075, #967, #1029, #1052, #863, #722

And that's only the currently active tickets.

Instead of being careful and working in a branch, now we usually break the trunk for a few days or weeks while implementing things.

[Hans Dembinski]

I am in favor of this, many good arguments were mentioned in the call on Mar 16, 2016. It makes working with the trunk easier and more standard, which also helps new users who just read how to do something with svn on the internet and wonder why it doesn't work. I understand why svn:externals are used in IceCube?, but it has always felt awkward to me (I am from Auger, we used a system like the one Claudio proposes).

One of the arguments in the call was about the needs of the many (normal users) versus the needs of the few (power users). Debates like this quickly go down a slippery slope, but the point was been made, so...

I am personally a big fan of making things easy/obvious for the general user, that's the philosophy I followed with my changes to steamshovel. Yet, I also want my work as a power user to not be hindered unnecessarily. The power users drive the software development of the experiment forward. They make new things that make new stuff possible, which leads to more papers, more grant money, etc.

My understanding is that we can make the work of the power user easier with this change, with a moderate impact on the general user. Yes, some people will get confused, but with the software channel in Slack we have an well-working help desk in place which will provide quick relief for most scratched heads. Having a system like Slack is a real advantage in this case to distribute information like this.

So, I think that the benefits outweigh the issues.

[Alex Olivas]

Replying to david.schultz:

Tickets that would benefit from easy repository-wide branching: #985, #1538, #1002, #1075, #967, #1029, #1052, #863, #722

And that's only the currently active tickets.

Instead of being careful and working in a branch, now we usually break the trunk for a few days or weeks while implementing things.

Some of those tickets are mine and I'm not seeing how this proposal is going to help me. I'm going to work from the trunk of combo and commit only when all tests pass, which is the model we've always advocated. That being said, the new model isn't going to hinder me either.

[David Schultz]

Replying to olivas:

Replying to david.schultz:

Tickets that would benefit from easy repository-wide branching: #985, #1538, #1002, #1075, #967, #1029, #1052, #863, #722

And that's only the currently active tickets.

Instead of being careful and working in a branch, now we usually break the trunk for a few days or weeks while implementing things.

Some of those tickets are mine and I'm not seeing how this proposal is going to help me. I'm going to work from the trunk of combo and commit only when all tests pass, which is the model we've always advocated. That being said, the new model isn't going to hinder me either.

I would argue that tickets like these would benefit from branching and saving intermediate work to the branch. It's hard to collaborate if we only commit to trunk when done. Also, checkpointing/backup is useful on major features (or any work that takes more than a few days).

[Claudio Kopper]

The new structure could also benefit a git-svn workflow if anyone wants to use that for their work (i.e. you would collect commits on your machine and then push them to SVN once done). Not sure this is a major reason to change anything, but it was pointed out on Slack and I think it should be documented here.

[David Schultz]

A note that non-svn solutions (a dev repository synced with the main one) are options.

nega from slack:

anyway to expand on my comment on the call... if there is something that will address the issues at hand and doesnt use svn that should be discussed in the tickets...

my point is, we shouldn't just focus on svn as the solution

[Alex Olivas]

Replying to claudio.kopper:

The new structure could also benefit a git-svn workflow if anyone wants to use that for their work (i.e. you would collect commits on your machine and then push them to SVN once done). Not sure this is a major reason to change anything, but it was pointed out on Slack and I think it should be documented here.

I already do this with our current structure and it works just fine.

[Alex Olivas]

Replying to david.schultz:

Tickets that would benefit from easy repository-wide branching: #985, #1538, #1002, #1075, #967, #1029, #1052, #863, #722

And that's only the currently active tickets.

Instead of being careful and working in a branch, now we usually break the trunk for a few days or weeks while implementing things.

What we've asked developers to do repeatedly is to make sure you build locally and that all tests pass for them. Developer due diligence is independent of the repository layout. They're still going to have to do this. If they can't/won't ensure it builds and all tests pass, it's unlikely they're going to branch and then ensure the branch builds and all tests pass before re-merging back to the trunk. This just sandwiches two extra steps around something some developers aren't doing anyway.

[Alex Olivas]

Replying to david.schultz:

Replying to olivas:

Replying to david.schultz:

Tickets that would benefit from easy repository-wide branching: #985, #1538, #1002, #1075, #967, #1029, #1052, #863, #722

And that's only the currently active tickets.

Instead of being careful and working in a branch, now we usually break the trunk for a few days or weeks while implementing things.

Some of those tickets are mine and I'm not seeing how this proposal is going to help me. I'm going to work from the trunk of combo and commit only when all tests pass, which is the model we've always advocated. That being said, the new model isn't going to hinder me either.

I would argue that tickets like these would benefit from branching and saving intermediate work to the branch. It's hard to collaborate if we only commit to trunk when done. Also, checkpointing/backup is useful on major features (or any work that takes more than a few days).

Branching projects isn't hard. Branching meta-projects and changing externals to point to the project branches isn't that hard either. I'd be happy to write a tutorial on how to do this. Granted this proposal would make this easier, but if there's truly an extensive re-work that spans many projects, the process of branching projects and then a meta-project and updating the externals is a small perturbation on the process.

[Alex Olivas]

As a counter proposal that addresses what I think are most of the relevant wishlist items is a variation on Claudio's "stable trunk" model.

​http://code.icecube.wisc.edu/svn/meta-projects/combo/stable

This uses the trunk of all projects, but with two fixed revision numbers, one each for the icetray and icecube repos.

Every morning (Mon-Fri) if the bots are green across the board I'll advanced the revision numbers for that meta-project. I can provide simulation and icerec meta-projects too if people want.

This doesn't making meta-project branching easier, but we can provide a script to do that if people want.

[David Schultz]

Replying to olivas:

Replying to david.schultz:

Replying to olivas:

Replying to david.schultz:

Tickets that would benefit from easy repository-wide branching: #985, #1538, #1002, #1075, #967, #1029, #1052, #863, #722

And that's only the currently active tickets.

Instead of being careful and working in a branch, now we usually break the trunk for a few days or weeks while implementing things.

Some of those tickets are mine and I'm not seeing how this proposal is going to help me. I'm going to work from the trunk of combo and commit only when all tests pass, which is the model we've always advocated. That being said, the new model isn't going to hinder me either.

I would argue that tickets like these would benefit from branching and saving intermediate work to the branch. It's hard to collaborate if we only commit to trunk when done. Also, checkpointing/backup is useful on major features (or any work that takes more than a few days).

Branching projects isn't hard. Branching meta-projects and changing externals to point to the project branches isn't that hard either. I'd be happy to write a tutorial on how to do this. Granted this proposal would make this easier, but if there's truly an extensive re-work that spans many projects, the process of branching projects and then a meta-project and updating the externals is a small perturbation on the process.

Maybe it's just me, but if I have to do svn copy more than twice I get highly annoyed and will just give up on it. So it might not be a technical hurdle, but a mental one.

[Alex Olivas]

Replying to david.schultz:

Replying to olivas:

Replying to david.schultz:

Replying to olivas:

Replying to david.schultz:

Tickets that would benefit from easy repository-wide branching: #985, #1538, #1002, #1075, #967, #1029, #1052, #863, #722

And that's only the currently active tickets.

Instead of being careful and working in a branch, now we usually break the trunk for a few days or weeks while implementing things.

Some of those tickets are mine and I'm not seeing how this proposal is going to help me. I'm going to work from the trunk of combo and commit only when all tests pass, which is the model we've always advocated. That being said, the new model isn't going to hinder me either.

I would argue that tickets like these would benefit from branching and saving intermediate work to the branch. It's hard to collaborate if we only commit to trunk when done. Also, checkpointing/backup is useful on major features (or any work that takes more than a few days).

Branching projects isn't hard. Branching meta-projects and changing externals to point to the project branches isn't that hard either. I'd be happy to write a tutorial on how to do this. Granted this proposal would make this easier, but if there's truly an extensive re-work that spans many projects, the process of branching projects and then a meta-project and updating the externals is a small perturbation on the process.

Maybe it's just me, but if I have to do svn copy more than twice I get highly annoyed and will just give up on it. So it might not be a technical hurdle, but a mental one.

Yeah. I don't have anything that addresses that. :)

[Alex Olivas]

Please set a milestone.

[Claudio Kopper]

updated version of the script taking into account the new serialization project in icetray

[Alex Olivas]

Pulling this off the shelf. Nathan has a plan and we're going to need to find a way to support meta-projects which pull from both svn and github, if we can and it's the best long-term solution. Otherwise we're stuck with the sub-optimal model of svn copies of tagged github projects (e.g. PROPOSAL).

[Nathan Whitehorn]

Current example tree is at svn/highlander/trunk and preserves all history except that for things in the icetray repository from 2009-2018 (restoring which is both optional and requires administrative intervention).

Use with svn is as normal, except much faster. In any case anyone wants to try git with it, this is the workflow for using git:

To clone:
git svn -s --no-minimize-url clone --revision 165886:HEAD ​http://code.icecube.wisc.edu/svn/highlander

To pull:
git svn rebase

To push:
git svn dcommit

All other git commands are as normal.

[Alex Olivas]

I broke this out into separate tickets, since several, and I believe, independent issues came out of this discussion. I'll work on merging some of the info here into the other tickets, but this ticket will still be around, even though technically it'll be closed.

[Alex Olivas]

Milestone Winter Solstice 2018 deleted