[#5284] Move to subversion for Horde code repository
Summary Move to subversion for Horde code repository
Queue Horde.org Servers
Type Enhancement
State Rejected
Priority 2. Medium
Owners
Requester chuck@horde.org
Created 2007-04-22 (4831 days ago)
Due
Updated 2008-06-13 (4413 days ago)
Assigned 2007-10-30 (4640 days ago)
Resolved 2008-06-13 (4413 days ago)
Milestone
Patch No

Comments
Chuck Hagenbuch <chuck@horde.org> 2007-04-22 01:44:03
I'm formally proposing migrating from CVS to Subversion for Horde.



I know there are some concerns about organizing applications and 
packages appropriately. I think that we might be able to use 
svn:externals to address some or all of those concerns. There are some 
useful notes about svn:externals here: 
http://weierophinney.net/matthew/archives/132-svnexternals.html

Chuck Hagenbuch <chuck@horde.org> 2007-10-30 20:54:20
Assigning to Ben to update when a dump is available from svn.horde.org 
(now a dns alias for nazik). We'll then need to make sure chora works 
appropriately, access control works or is ported, and the pre/post 
commit scripts run.

Ben Klang <bklang@horde.org> 2007-10-31 18:27:22
The Subversion repository has been populated with copy of CVS from 
last week.  It's no good for checking in new code, but we can use it 
as a proof-of-concept and to tweak the pre-/post-commit scripts.  I 
would suggest we use the dav_svn mechanism instead of ssh like we do 
today as it allows us greater control over ACLs and fewer OS tweaks 
for supporting virtual users.



For future reference, it will take approximately 3 - 4 hours for the 
scripts to run when we do the "live" conversion, so we'll need to 
schedule the time to make sure no one needs to make commits while it's 
running.



The repository lives at nazik:/usr/local/horde/svn

Chuck Hagenbuch <chuck@horde.org> 2007-11-01 04:19:54
http://svn.horde.org/trunk/



Still to do:



* where do we want chora (assuming we prefer that in general for the 
web frontend, but we need to differentiate the URLs then)



* do we want to require ssl for authenticated svn?



* port our access control hierarchy



* port our pre-commit checks (php-lint)



* port post-commit checks (or replace with another commit diff mailing 
list - do we want to maintain the Chora links, or go with inline 
diffs, or ... ?)



I think that's actually it.

Chuck Hagenbuch <chuck@horde.org> 2007-11-01 05:28:25
I have a (only tested with badly mocked-up inputs) lint script in 
/usr/local/horde/svn/hooks/commit-php-syntax-check.pl, enabled in the 
pre-commit script.

Jan Schneider <jan@horde.org> 2007-11-01 10:49:29
IMO:



> * where do we want chora (assuming we prefer that in general for the

> web frontend, but we need to differentiate the URLs then)



If possible we should try to also run it on http://svn.horde.org. I 
guess we don't need the svn root folder for svn access and can use 
some mod_rewrite or similar magic to correctly redirect requests to 
either svn or chora. If not possible, svnweb.horde.org should work too.



> * do we want to require ssl for authenticated svn?



Yes.



> * port post-commit checks (or replace with another commit diff

> mailing list - do we want to maintain the Chora links, or go with

> inline diffs, or ... ?)



I'm not sure what you mean with that.

Chuck Hagenbuch <chuck@horde.org> 2007-11-02 03:44:13
>> * port post-commit checks (or replace with another commit diff

>> mailing list - do we want to maintain the Chora links, or go with

>> inline diffs, or ... ?)

>

> I'm not sure what you mean with that.



Right now cvs@lists.horde.org gets the log message, list of changed 
files, and links to chora. We could also do inline unified diffs, 
colorized diffs, or some combination thereof.



Another idea I'd like to implement is to recognize #\d+ (# followed by 
numbers) in log messages and have a post-commit script that posts the 
log message and a link to the changeset to any bug mentioned in a 
commit.

Chuck Hagenbuch <chuck@horde.org> 2007-11-02 03:51:08
> If possible we should try to also run it on http://svn.horde.org. I

> guess we don't need the svn root folder for svn access and can use

> some mod_rewrite or similar magic to correctly redirect requests to

> either svn or chora. If not possible, svnweb.horde.org should work

> too.



I'm not sure what we would rewrite on? We could do something like 
http://svn.horde.org -> Chora; http://svn.horde.org/repository -> 
readonly dav_svn, https://svn.horde.org/repository - developer 
dav_svn. We just couldn't then put something named "repository" into 
the top level of svn. :) Suggestions for something other than 
"repository" are fine by me too, but that's probably the way to go.



One more thing to consider: do we want to keep the structure ported 
from cvs, with a single trunk and global tags and branches 
directories? Or do we want one set of trunk/tags/branches per project? 
Keeping the existing structure is simplest, and while it's a lot 
easier to move stuff around with svn, it's not *completely* free 
(still grows the repository a bit).

Chuck Hagenbuch <chuck@horde.org> 2007-11-02 03:59:31
On authorization:



See 
http://svnbook.red-bean.com/en/1.4/svn-book.html#svn.serverconfig.pathbasedauthz



I think it'll be trivial to port our "avail" file to authz. The 
question is, do we need to and want to? I think I definitely want to 
push for removing the distinction between "core" and "general" 
developers. There is little difference, and it probably mostly 
discourages people lumped into "general" (not so much now as it's just 
2 people at the moment) from working on the website (hordeweb).



That gets rid of the need for most rules; after that it's just 
granting access to a few paths to a few developers. But see the yellow 
box on that subversion page; I am currently inclined to agree and to 
say that we don't really need to sandbox people off. If there are 
social problems, then we probably shouldn't be letting people commit 
code, period.

Jan Schneider <jan@horde.org> 2007-11-02 10:53:12
> That gets rid of the need for most rules; after that it's just

> granting access to a few paths to a few developers. But see the

> yellow box on that subversion page; I am currently inclined to agree

> and to say that we don't really need to sandbox people off. If there

> are social problems, then we probably shouldn't be letting people

> commit code, period.



I generally agree, especially for such a small project like ours. 
Though I have always considered the restrictions rather as a 
prevention. Remember the PEAR developer tagging the whole PHP CVS 
server a few years ago? This kind of stuff.

Beside that it still gave me that warm fuzzy feeling when adding new 
developers I don't know well to cvs. But it's questionable if this is 
worth it.

Jan Schneider <jan@horde.org> 2007-11-02 10:56:59
>>> * port post-commit checks (or replace with another commit diff

>>> mailing list - do we want to maintain the Chora links, or go with

>>> inline diffs, or ... ?)

>>

>> I'm not sure what you mean with that.

>

> Right now cvs@lists.horde.org gets the log message, list of changed

> files, and links to chora. We could also do inline unified diffs,

> colorized diffs, or some combination thereof.



I *hate* colorized inline diffs. I can see some use for unified diffs 
as attachments, or even colorized diffs as attachments (I have seen a 
few that really were unified diff, just colored) but I wonder if it's 
worth the message bloat. I have no strong opinion about this, but I'm 
definitely happy with the way it works now.



> Another idea I'd like to implement is to recognize #\d+ (# followed

> by numbers) in log messages and have a post-commit script that posts

> the log message and a link to the changeset to any bug mentioned in a

> commit.



Yep.

Jan Schneider <jan@horde.org> 2007-11-02 11:04:31
>> If possible we should try to also run it on http://svn.horde.org. I

>> guess we don't need the svn root folder for svn access and can use

>> some mod_rewrite or similar magic to correctly redirect requests to

>> either svn or chora. If not possible, svnweb.horde.org should work

>> too.

>

> I'm not sure what we would rewrite on? We could do something like

> http://svn.horde.org -> Chora; http://svn.horde.org/repository ->

> readonly dav_svn, https://svn.horde.org/repository - developer

> dav_svn. We just couldn't then put something named "repository" into

> the top level of svn. :) Suggestions for something other than

> "repository" are fine by me too, but that's probably the way to go.



I'm not sure it this would work out of the box or if we need some 
rewriting rules for that. If it works, even better!



> One more thing to consider: do we want to keep the structure ported

> from cvs, with a single trunk and global tags and branches

> directories? Or do we want one set of trunk/tags/branches per

> project? Keeping the existing structure is simplest, and while it's a

> lot easier to move stuff around with svn, it's not *completely* free

> (still grows the repository a bit).



I can see advantages on both approaches. I guess we have to collect 
some pros and cons, especially technical ones. Separate projects 
sounds nice, but since we copy/move code around between projects from 
time to time, I can see having all in one (svn) project making more 
sense.

But it's more important whether one of these setups makes it easier to 
checkout one (horde) module inside another.

Chuck Hagenbuch <chuck@horde.org> 2007-11-04 04:27:52
Status update:



anonsvn is available from http://svn.horde.org/repository/

developer svn is available from https://svn.horde.org/repository 
(though right now only I have a password :)

chora is available from http://svn.horde.org/



On commit emails:

I think inline text-only diffs with a max message size would be ideal. 
I agree about colored diffs and I also really hate systems that send 
out commit messages over a megabyte. I think if we can do it, 
something that sends a main body part with the log message and chora 
links, and then inline attachments of type text/diff for up to 40k or 
so of changes, seems nice.



On checking modules out inside other modules:

The svn way to do this is with externals, BUT: there is no way to 
commit to everything at once using externals (per 
http://svn.haxx.se/users/archive-2007-01/0955.shtml). The ideal 
situation here is that we would have a tree of apps that didn't need 
to be in the webroot, so we wouldn't need to put them inside each 
other. Externals is a reasonable solution that would let us have a dev 
tree and just commit to modules separately. Another option is 
symlinks, which subversion can handle now - we could put links to the 
apps inside the horde/ module, assuming they were now on the same 
level as horde.



One side effect of having different developer and anonymous URLs 
(https vs http) is that if we want to use externals on our own stuff, 
it has to be different sets of externals properties for dev and anon.



On commit hooks:

I haven't done anything about a new commits list. The pre-commit hooks 
are not working yet.



On access control:

Unless someone feels strongly I think we should try authenticated == 
write access at first.

Chuck Hagenbuch <chuck@horde.org> 2007-11-04 20:48:49
One other possibility that occurred to me at church this morning :) - 
we could move all FW_3 apps under the FW_3 horde in the repository, so 
that they are useable as-is. Then we could re-engineer HEAD to simply 
work while expecting things to be side-by-side. That way we get 
reasonable behavior for the (not so huge) set of FW_3 apps, and can 
break BC for HEAD and have a nicer structure overall, period.



In other news I've figured out what was wrong with the pre-commit hook 
and it's about halfway fixed now.

Chuck Hagenbuch <chuck@horde.org> 2007-11-05 02:55:12
pre-commit hook (for php syntax checking) is now working. it's in the 
hooks/ dir of the repository and enabled.

Jan Schneider <jan@horde.org> 2007-11-05 11:06:30
> On commit emails:

> I think inline text-only diffs with a max message size would be

> ideal. I agree about colored diffs and I also really hate systems

> that send out commit messages over a megabyte. I think if we can do

> it, something that sends a main body part with the log message and

> chora links, and then inline attachments of type text/diff for up to

> 40k or so of changes, seems nice.



Sounds good.



> On checking modules out inside other modules:

> The svn way to do this is with externals, BUT: there is no way to

> commit to everything at once using externals (per

> http://svn.haxx.se/users/archive-2007-01/0955.shtml). The ideal



And this has always been the show stopper for me to switch Horde to 
SVN. I had hopes that this could properly be dealt with in the meantime.



> situation here is that we would have a tree of apps that didn't need

> to be in the webroot, so we wouldn't need to put them inside each

> other. Externals is a reasonable solution that would let us have a

> dev tree and just commit to modules separately. Another option is

> symlinks, which subversion can handle now - we could put links to the

> apps inside the horde/ module, assuming they were now on the same

> level as horde.

>

> One side effect of having different developer and anonymous URLs

> (https vs http) is that if we want to use externals on our own stuff,

> it has to be different sets of externals properties for dev and anon.



That sounds awkward.

Jan Schneider <jan@horde.org> 2007-11-05 11:11:18
> we could move all FW_3 apps under the FW_3 horde in the repository,

> so that they are useable as-is. Then we could re-engineer HEAD to

> simply work while expecting things to be side-by-side. That way we

> get reasonable behavior for the (not so huge) set of FW_3 apps, and

> can break BC for HEAD and have a nicer structure overall, period.



But that would meen that anybody checking out the horde module from 
FW_3 would get all apps checked out too, correct?



After taking a look at the current SVN repository, I changed my mind 
and think we should setup separate modules for each Horde app. Having 
all tags and branches for all horde modules ever in branches/ and 
tags/ looks really cluttered. I think it's worth not being able to svn 
mv between horde modules for the benefit of a cleaner layout.

Chuck Hagenbuch <chuck@horde.org> 2007-11-05 17:17:30
> But that would meen that anybody checking out the horde module from

> FW_3 would get all apps checked out too, correct?



Right - I don't think that's so bad. If they want exact installs they 
can use tarballs, and the world of FW_3 apps isn't as huge as HEAD apps.



> After taking a look at the current SVN repository, I changed my mind

> and think we should setup separate modules for each Horde app. Having

> all tags and branches for all horde modules ever in branches/ and

> tags/ looks really cluttered. I think it's worth not being able to

> svn mv between horde modules for the benefit of a cleaner layout.



We could still svn mv/copy between them, unless we really made them 
completely separate repositories. There is nothing hardcoded in 
subversion about trunk/tags/etc - it's just a convention. We could to 
a top-level list of modules with their own trunk, tags, branches dirs, 
or we could do subdirs inside the /tags and /branches dirs just like 
we have inside /trunk.

Jan Schneider <jan@horde.org> 2007-11-05 17:45:05
>> After taking a look at the current SVN repository, I changed my mind

>> and think we should setup separate modules for each Horde app. Having

>> all tags and branches for all horde modules ever in branches/ and

>> tags/ looks really cluttered. I think it's worth not being able to

>> svn mv between horde modules for the benefit of a cleaner layout.

>

> We could still svn mv/copy between them, unless we really made them

> completely separate repositories. There is nothing hardcoded in

> subversion about trunk/tags/etc - it's just a convention. We could to

> a top-level list of modules with their own trunk, tags, branches

> dirs, or we could do subdirs inside the /tags and /branches dirs just

> like we have inside /trunk.



I would prefer top-level modules, then trunk/tags/branches. But we 
should make sure first that Chora works fine with that. :)

Matt Selsky <selsky@columbia.edu> 2007-11-05 18:23:14
> I would prefer top-level modules, then trunk/tags/branches. But we

> should make sure first that Chora works fine with that. :)



Does that make it more of a pain to check out all HEAD/trunk apps and 
test them together?  Or all FW_3?  I don't want to have to check out 
each app individually...

Chuck Hagenbuch <chuck@horde.org> 2007-11-05 20:43:06
> Does that make it more of a pain to check out all HEAD/trunk apps and

> test them together?  Or all FW_3?  I don't want to have to check out

> each app individually...



You could do that with svn:externals refs.

Chuck Hagenbuch <chuck@horde.org> 2007-11-05 21:00:23
> I would prefer top-level modules, then trunk/tags/branches. But we

> should make sure first that Chora works fine with that. :)



I did a little of this with Horde and IMP for testing purposes. For 
example I moved some of the current ALPHA tags over:



http://svn.horde.org/repository/imp/tags/



In Chora:

http://svn.horde.org/imp/tags/IMP_4_2_ALPHA/

Matt Selsky <selsky@columbia.edu> 2007-11-05 22:09:08
> You could do that with svn:externals refs.



But no check-ins to externals...

Chuck Hagenbuch <chuck@horde.org> 2007-11-05 22:15:35
> But no check-ins to externals...



Sure you can. Just not automatic (recursive) commits.

Jan Schneider <jan@horde.org> 2007-11-05 23:31:20
> We could still svn mv/copy between them, unless we really made them

> completely separate repositories.



We should even consider that. I don't know but I somehow don't feel 
comfortable with 6-digit version numbers that we'll get some day.

Chuck Hagenbuch <chuck@horde.org> 2007-11-06 04:51:13
>> We could still svn mv/copy between them, unless we really made them

>> completely separate repositories.

>

> We should even consider that. I don't know but I somehow don't feel

> comfortable with 6-digit version numbers that we'll get some day.



I think that that's just making us things hard for ourselves, and we 
might get to 6-digit numbers anyway someday (okay, less likely since 
we're only about 60% there with all horde commits so far, but that 
means it'll take us a while to get there even with the whole 
repository anyway). Not feeling comfortable with the numbers just 
doesn't seem like a good reason to me.

Michael Slusarz <slusarz@horde.org> 2008-06-13 06:53:26
Should we keep this ticket open, especially given the Wiki entry at 
http://wiki.horde.org/Project/FutureSCM and the (likely) decision that 
we aren't going to use SVN at all?

Jan Schneider <jan@horde.org> 2008-06-13 08:15:09
No, one place to track this is sufficient.