Build tool for mixed Clojure/Java projects

1) Must handle Clojure namespaces that are AOT-compiled.
2) Must handle Clojure namespaces that are not AOT-compiled.
3) Must handle Java source code files.
4) Must handle dependencies in the form of on-disk jar files (not in
any repository)
5) No XML configuration files.

Candidates that I have tried and found insufficient include
- Leiningen (no dependencies without repositories)
- Cake (doesn't handle Java source files)
- Eclipse/Counterclockwise (doesn't handle AOT compilation)
- ant, maven: XML configuration files
- scons: incomplete Java support, no Clojure support

Is there anything else worth trying?

Thanks in advance,
Konrad.

Speaking for zero actual experience:
Have you tried Gradle?
http://www.gradle.org/

// ben

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

Speaking with a bit of experience: gradle has a check for all of your points, although clojuresque doesn't handle mixed 1) and 2) points. At the moment it's “either or”. Since you are screwed anyway when you have to use AOT-compilation. But I'm open for arguments why one would want to mix points 1) and 2).

Sincerely
Meikel

> 4) Must handle dependencies in the form of on-disk jar files (not in
> any repository)

For the record, leiningen can do this by adding a repository with a
file:/// URL; it's just not documented because I think it's a bad idea.

-Phil

but we use features 2,3,4,5 on your list easily. the project has
java, clojure and
groovy code. it just works.

--
Omnem crede diem tibi diluxisse supremum.

Why?

One pillar of Clojure philosophy is to get out of the way and let
programmers get the job done, with a minimum of fuss and ceremony.
Unlike Java.

Requiring any project that has dependencies, even if these are only
other local projects, have a repository adds gratuitous ceremony.

And this is especially true if the user has little or no prior
experience setting up and working with repositories, as they grow
their programming skillset. Ordinarily, they would gradually progress
from single-file projects to multi-file projects to projects with
common, locally-installed third-party libraries and/or other local
projects as dependencies, and eventually to projects that use and are
hosted on repositories of some sort. Your "I think it's a bad idea"
amounts to thinking that forcing the third and fourth steps in that
progression to be combined into one single giant leap is a *good*
idea, and I'm rather dubious of any such claim. One new source of
complexity should be mastered at a time, in my view, absent a *very*
good reason, and repositories are definitely a humongous source of
complexity, so an especially strong case can be made that that
*particular* new source of complexity should be tackled separately
from any other.

(Just consider: starting out using repositories requires mastering a
whole new client/server app type on a par with email, newsgroups, ftp,
and the web; then there's all these different hosting alternatives --
maven, github, sourceforge, google code, etc. and sometimes software
of the same name or closely related like maven and git, as well as
other software like rcs, cvs, and svn; then there's the associated
terminology: commits, pull requests, push requesus, branches, masters,
forks, checkouts and checkins, and so forth, not all of them
applicable to each individual variation on the theme; then there's
individual projects' often-partly-unwritten rules regarding forking,
commit access, pull/push requests, and what-have-you; and setting up a
local repository of one's own means installing a server, securing it,
installing a client, learning how to use it, and so on and so forth.

Indeed, a lot of people will never before have installed and secured a
server of any kind in their entire lives, I expect; that itself is
something whose learning I think should probably be tackled
separately, but if you haven't already done something like run your
own listserv or web server (not just site, you own httpd process which
if hacked means your own computer is violated), then starting to use
local version control means tackling that at the same time as a load
of other new and moderately complex things. To which your
recommendation adds a third group of complicated new things. So, it's
not even two *steps* combining into one giant leap; it's a step
followed by an already-fairly-large *leap* combining into one
*ridiculous* leap.)

--
Protege: What is this seething mass of parentheses?!
Master: Your father's Lisp REPL. This is the language of a true
hacker. Not as clumsy or random as C++; a language for a more
civilized age.

OK...ridiculous leap reduced to merely huge leap then, if you only
have to learn 2 of these three things at once: big, interdependent
groups of projects, maven/etc. repository-accessing tools, and server
administration. :)

Encouraging "works on my machine" builds is by definition antithetical to the very idea of build automation. Leiningen is not a "yes tool".

-Phil

That presupposes that nobody will ever use leiningen for any case
that's intermediate between, or orthogonal to, "personal project,
single file, no unusual deps, just load in REPL in IDE and use" and
"publicly visible large open source project with many deps that
everyone and his great-aunt should be able to build". For example,
"personal project, many files and local/third party deps" is not
covered and neither is "closed-source project for
small-to-medium-sized company without own version control server". For
the former, and arguably the latter, anything involving the word
"repository" in any way shape or form strikes me as
potentially-unnecessary ceremony. (Larger businesses will presumably
have in-house version control and keep proprietary code bases in a
repository, so there's no issue there. Also, larger businesses can
generally just buy their way out of any problem, inconvenience, or
excess of ceremony by hiring an extra person or two to deal with it.
:) Meanwhile, large FOSS projects all seem these days to have public
version control repositories, either freely provided by sites like
github and sourceforge or, in the most high profile and especially
business-backed cases, their very own server farms with project web
sites and repository.)

And this is how leiningen making users jump through hoops to do fairly
simple builds with local dependencies *itself* can encourage "works on
my machine" builds. :) Which is preferable from that stand point,
after all: a leiningen build script or lots of idiosyncratic shell
scripts?

Off the top of my head:

(ns leiningen.sub
(:use [leiningen.core :only [apply-task task-not-found]]))

(defn sub [task-name & args]
(doseq [project (projects-in-dir)]
(apply-task task-name project args task-not-found)))

Implementation of projects-in-dir left as an exercise for the
reader. Shouldn't be more than four lines. Push the lein-subs plugin to
clojars: problem solved in a repeatable, maintainable way.

-Phil

Thanks for everyone's suggestions and comments!

Some remarks:

Gradle/clojuresque: I never tried them, but it looks like I should!

AOT compilation: My case is probably a bit complex in this respect. I
need AOT compilation only because I want to produce an executable jar,
so there is at least one namespace that must be AOT compiled. For
building that executable jar, it doesn't matter if everything else is
AOT compiled as well. But I want to be able to build a standard
library jar with just the Clojure code to make sure it works with
future Clojure releases. One solution would be to have two completely
different builds, one for the executable jar and one for the library
version, but using the same set of source code files.

With absolute filenames in a URL, it's indeed a bad idea because the
project becomes specific to a machine/installation. What I'd like to
see instead is a project-local repository, e.g. a "deps" folder with
jar files and source code directories.

There are several reasons why I want dependencies outside of
repositories:

1) Simplicity. I want to distribute a single tar file to collaborators
(who are new to Clojure and the JVM world) containing all dependencies
and build scripts. I want to tell them "install <name of build tool>,
take this file, play with the source code, and just type <build-
command> to build an executable". They will struggle with a new
language and a new environment already, there's no need to add the
complexities of repositories. To make it worse, we all live behind
proxies that require messy configuration for tools like Maven.

2) Network independence. I often work without Internet access, and I
don't want to be blocked at some point because some build tool wants
to access some repository to see if my version is still current.

3) Confidentiality. Sometimes I work with unpublished code that I
promised not to give to anyone else. I can't keep that promise if the
code leaves my machine in any way. For that case, the file:///
repository would of course be fine.

Having used cake in the past for simpler projects, I know that it can
handle project-local dependencies. Just add a classpath setting
to .cake/config (I'd have to look up the exact name of that variable).
If cake can indeed handle Java code, then it might be just fine for my
needs.

Being Maven, this would still require repositories, right? Even a
local Maven repository is more than I'd like to impose on my
collaborators.

I'd prefer one build system to two separate layers, one of which I'd
write myself. Moreover, most of my dependencies are not projects, but
simply jar files picked up from various sources.

Thanks again to everyone! I'll look at Java with cake, and also at
Gradle.

Konrad.

http://stackoverflow.com/questions/2572811/gradle-make-a-3rd-party-jar-available-to-local-gradle-repository

and getting started is at https://bitbucket.org/kotarak/clojuresque/wiki/Home

while the tutorial is written for gradle 1.0.x milestone releases. I
use gradle 0.9.2,
and it works fine for me.

> play with the source code, and just type <build-command> to build an

--

You seem to want to make two things: a library and an application. For
that, I think it's only proper if you have two build scripts, since
you have two build artifacts. Indeed, the application may be better
off depending on the library than directly incorporating its source
code as some of its own.

> play with the source code, and just type <build-command> to build an

This is what I meant about avoiding unnecessary ceremony.

The above provide some reasons why the build process should not
*require* an internet connection if all dependencies are already
available locally. I don't have a problem if some build tool checks
for updates, so long as it treats a failed check the same as finding
there's no update, give or take a non-fatal warning or two.

On the other hand, the result on actually finding an update should
probably be a user prompt in interactive mode and a build without the
update in batch mode. Otherwise the update, if it breaks backward
compatibility, could break the build. A build that worked yesterday
and breaks today without one having changed any local files, neither
the build script nor the sources nor anything else, is going to be a
head-scratcher to deal with, and in many contexts time is money.

Presumably also for network-independence and avoiding annoying
proxies. But if it needs an absolute path, that may* break the "just
tar up the source tree and mail it to someone" capability and forces
even small collaborative projects to set up and use some kind of code
repository instead. As I've argued elsewhere, repository use scales
poorly at the small-project end, unsurprisingly since repositories
were specifically invented as a tool to make large-scale projects with
huge numbers of collaborators and vast reams of code manageable. Using
a high-falutin' version control system for 12kloc and three developers
is like using a bazooka to swat a fly, a top-of-the-line Oracle
database product on expensive iron to track your personal expenses and
balance your checkbook, or hashmaps instead of arraymaps to store
little three-key objects that number in the tens of thousands. :)

* Unix filesystems seem to be standardized enough in layout that using
absolute paths of \usr\src\whatever and suchlike may be viable.
Except that it won't work for a collaborator whose machine runs
Windows, and, worse, it requires working as root or else letting
unprivileged users create executables in world-readable places -- ick.

Indeed, Maven-based stuff seems oriented around the assumption that
any third-party dependencies you will have, in particular, will
generally be publicly hosted on Maven repositories themselves rather
than just a ten-cent web site or Blogspot article or Facebook page
with a .jar download link. Another case of not scaling well at the
low-size end of things, or at the shallow end of the skill pool or the
low end of the budget scale for that matter. (Setting up a Facebook
profile or a blog on an existing service: free and easy. Uploading a
jar file you built to Rapidshare or some other such site and posting
the link: free and easy. Setting up some kind of publicly-visible code
repository: expensive and complex. Using an existing repository
service site: perhaps free and perhaps less complex, but still fairly
complex.) But this time biting the small-time library's users rather
than its small-time developer...

That's actually a bit worrying. I'm not sure I want a potential
security hole into my computer, such as a repository, being "handled
for me" without having *some* input into the matter. For example if I
intend to use it purely locally I'd want to firewall the port it's
using to make sure it's unreachable from any address other than
127.0.0.1.

On the one hand, manually managing stuff like that is hard, and it's
complex to learn the first time.

On the other, making it *too* simple, at least when it has
implications for the machine's security, is not generally a good
thing.

There's also the fact that everything remains simple (maybe too
simple; see above) only so long as you remain inside the cushioned box
of the particular IDE. As soon as you try to do anything else -- tweak
something from vi or Notepad, significantly refactor and rename and
restructure stuff in a manner that changes some file paths, move to or
sync with another machine, share with a collaborator, even just find
the source files to back them up somewhere safe -- you're in the deep
woods, possibly without a good map. :)

"Repository" need not imply anything to do with networking. I'm sure
someone will correct me if I'm wrong, but I am pretty sure that the
repository Steve is talking about above is just a hierarchy of files
in your home directory.

--
Michael Wood <esio...@gmail.com>

You might be able to use the :impl-ns option in gen-class to limit the
extent of AOT compilation.

--
Hugo Duncan

So, not actually a repository, then. :)

One problem with that is that you generally can't *not* use your IDE's
built in project management unless you don't use an IDE at all (vi,
emacs, Notepad, whatever rather than Eclipse, IDEA, Netbeans,
whatever).

That's not *too* bad, compared to Greenspun's 50%. On the other hand
I'd think the more natural comparison would be to make, which after
all long predates maven and may even be half as old as Common Lisp. :)

I was using it in the sense typically meant in phrases like "source
code repository", as seems reasonable given the context, but oh well.

Perhaps, but not by default. File->New->Project creates built-in
project management projects, at least in Eclipse, to judge by the fact
that the result appears to be a project to the built-in project
management functionality. :)

What method was used to create the projects ab initio? Some sort of
direct interaction with Maven, followed by some kind of import into
each IDE instance? Or can one of those IDEs create a Maven project (as
opposed to a built-in project management project) ex nihilo by ticking
some box somewhere?

(The third option, that one or each makes Maven projects natively by
default or even has no distinct "built-in project management project"
type, seems to be excluded by your earlier statement of "I'd be more

maven/lein/cake/etc. than if you're using the IDEs built in project
management facilities".)

If you're using git, "source code repository" could easily be local
and not require an Internet connection... so I think it depends on
your experience :)

In the OP comment, I certainly took repository to just mean "somewhere
you store stuff" and hence it could easily be local...
--
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/
Railo Technologies, Inc. -- http://www.getrailo.com/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

The same goes for CVS or Subversion or Mercurial and probably various
other "source code repositories". If you want to network them you
can, but you have to do something extra (e.g. run cvs-pserver or
mod_dav_svn or svnserve or hg serve etc.)

Things like Debian package archives are the same. They have a
particular structure and set of metadata and they may or may not be
available over the network (via FTP or HTTP). If you have one locally
(e.g. on CD) you can use it without setting up an FTP server or web
server, but of course then nobody else can use it from their machine.

I take it to me "somewhere you can store stuff in a particular
structure perhaps with metadata" so that maven knows what's there and
how to set up class paths and whatever else it needs to do with it.

--
Michael Wood <esio...@gmail.com>

> repository Steve [Lindsay] is talking about above is just a hierarchy of files
> in your home directory.

Right, there's a per-user repository at ~/.m2/repository .

Without any change to lein, its "install" subcommand will place a jar for the current project within ~/.m2/repository . The jar can be used from there in other local projects. Here's an example:

% cd <project-A-dir>
% lein install
[...]
[INFO] Installing ./projectA-1.0.0-SNAPSHOT.jar to ~/.m2/repository/projectA/projectA/1.0.0-SNAPSHOT/projectA-1.0.0-SNAPSHOT.jar
% cd <project-B-dir>
% lein install
[...]
[INFO] Installing ./projectB-1.0.0-SNAPSHOT.jar to ~/.m2/repository/projectB/projectB/1.0.0-SNAPSHOT/projectB-1.0.0-SNAPSHOT.jar
% cd <project-C-dir>
% cat project.clj
defproject projectC "1.0.0-SNAPSHOT"
:description "FIXME: write description"
:dependencies [[org.clojure/clojure "1.2.1"]
[projectA "1.0.0-SNAPSHOT"]
[projectB "1.0.0-SNAPSHOT"]])
% lein deps
Copying 3 files to <project-C-dir>/lib
% ls lib
clojure-1.2.1.jar projectB-1.0.0-SNAPSHOT.jar
projectA-1.0.0-SNAPSHOT.jar

This appears to meet Konrad's original specs except (depending on the intended meaning of "repository") for:

I would argue that the ~/.m2 repository is nearly as easy to work with as any other local, on-disk scheme one might envision and has the benefit of working with any maven-compatible tool.

It also works for arbitrary jars one may have on disk (acquired from any source) via:

http://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html

--Steve

I'd be very interested to know how one checks out a file from a CVS
repository without cvs-pserver running. You do a cvs checkout whatever
at the command prompt, the command interpreter runs the cvs client,
and the cvs client then connects to ??? (apparently not the
cvs-pserver you're not running) using ??? (apparently not cvs's wire
protocol over TCP/IP on the 127.0.0.1 loopback interface) to perform
the checkout ...

2011/7/5 Steve <stephen....@gmail.com>:

Unless there's a bug involved (and I suspect there's a rampant one
somewhere :) ), CCW handles AOT compilation.

Would it not handle it, I would not be able to release CCW itself !

Indeed, currently there are cyclic dependencies between java and
clojure code in CCW, and this is handled by Eclipse's Compiler only :
Eclipse compiles java files to (incompletely working) class files.
The CCW builder builds the missing AOT class files (thanks to
Eclipse's compiler having put the necessary signatures in place in the
java class files, even if the implementations are not complete yet).
After the CCW build, Eclipse refreshes the project and does a second
compile pass on its java files, ending the roundtrip with everything
being OK.

Sooo... how does CCW handle AOT compilation ?
Here's the current state of the art. It's subject to laarge
improvements, but I haven't opened this can of worms yet, I'm working
on other more urgent CCW topics.

So there it is :
a) Select your project's node in the Package Explorer
b) Trigger its contextual menu, select "Run as", select "Clojure Application"
I *insist* (*) : you must trigger the Run from the project's node.
Only with the project's node will the incremental compilation be
activated for the started process' JVM.
Then you should see:
1) A java console view for the underlying process' JVM inputs/outputs
2) A (n)REPL client view speaking with the underlying process' JVM via
either the nrepl dependency already present in your project's
classpath, either added "on the fly" to the classpath by the "Clojure
Application" launch configuration machinery (a clever ccw trick)
3) a "background" user job running (generally on the bottom right)
reporting some kind of progress related to the AOT compilation of the
project.
4) when the background user job has finished, you should see that your
project's "classes/" directory is full of AOT compiled class files.

Currently, there's one drawback : CCW does not refrain AOT compilation
from "compiling too much things", nor does it remove AOT compilation
artifacts that should generally not be desired (e.g. anything in
packages not being related to the project's own namespaces).

So now that you have the explanations, if this works for you as it
does for me, Eclipse+CCW will handle for you, out of the box:

(*) : In contrast, if you trigger a run of "Clojure Application" from
the contextual menu of a specific clojure lib file, the process will
not be started in "project mode", but in "file mode". Which implies
that CCW will just pre-load the selected file on the clojure process,
but do none of what it does when run in "project mode" : neither
attempt to AOT compile the whole project when the process starts,
neither attempt to (again) re-AOT compile the whole project everytime
you save a file in the editor.

HTH,

Cheers,

--
Laurent

2011/7/3 Konrad Hinsen <konrad...@fastmail.net>:

> Thanks in advance,
>  Konrad.

How would that be implemented, though? Without the server running to
enforce mutual exclusion and detect edit collisions and everything,
the whole notion of "checkin" and "checkout" appears to become
meaningless.

It doesn't "connect to" anything. It reads and writes the files
directly when used like this.

Exactly. And I've just tested it to be sure. Also, CVS originally
did not support networking at all. It was originally designed to be
used by different users on a multi-user machine.

Anyway, the point was that "repository" needn't have anything to do
with networking.

--
Michael Wood <esio...@gmail.com>

Sorry, but I think version control and, particularly, dealing with
edit collisions is not something you can solve as easily as just
slapping a lock onto each file access, or else version control servers
could just be FTP servers that locked their shared files during each
access.

And then there's the niggling little matter of Unix file locks being
advisory rather than mandatory. Do you really want proper resolution
of edit collisions (or, worse, the integrity of the under-the-hood
state maintained for the repository vis a vis making sense and being
valid data structures) to be entirely on the honor system? At
*minimum* it would seem that a real database with ACID and
transactions would be needed -- A to avoid race conditions (no
advisory locks here!), C to keep the internal state invariants valid,
I to be able to deal with edit collisions in a sane manner, and D for
the obvious reasons. And a suitable software front-end. And now we're
back to having at least one server in the mix, namely the DBMS at the
backend. :)

Here's a simple question to ask yourself whenever considering using
advisory file locks to guard complex on-disk data with invariants to
maintain: *If the power goes off halfway through some kind of
transaction, can the system recover without a lot of manual hacking to
patch up the state of the on-disk data structures? Will my losses be
limited solely to the one transaction in progress at the moment of the
outage?*

If you don't have a satisfactory answer there, then you probably need
a real database. Of course perhaps you are clever and can design your
own on-disk data structures that will fail-soft in some manner under
such circumstances and convince yourself beyond a reasonable doubt
that it'll work, but if not ...

If by "edit collisions" you mean merge conflicts, these are all done
on the client. Of course if two clients are trying to write to the
same file in the repository at the same time there could be problems,
so I do think that talking to a server over the network is best, but
CVS originally did not support this and still supports non-networked
access.

No, not meaningless. Just perhaps less reliable.

CVS does not support atomic commits either, which some people find
acceptable and others do not.

--
Michael Wood <esio...@gmail.com>

Maybe so, but that is all beside the point. The point was that
"repository" needn't mean "networked".

If the CVS clients are the only ones doing the locking, then why not?

This is why various people moved away from CVS. To get atomic commits etc.

Right, I agree it's best to have a server controlling access to the
repository, but that does not mean that the concept of a repository is
linked to having a server or a network.

Right, and this is a problem with CVS.

CVS is an old version control system that was far better than the
other things around at the time it was invented, but most people seem
to agree that other, more modern, version control systems are better
and/or more reliable.

--
Michael Wood <esio...@gmail.com>

Maybe just "repository that isn't fragile or outright broken" then. ;)

ARE they the only ones doing the locking OR otherwise accessing the
files, though?

No, just the concept of a reliable repository. ;)

Unsurprisingly. With apologies to Leonard McCoy:

"Non-atomic commits. Funduscopic examination. What is this, the Dark Ages?"

Perhaps, but I'd still say that even if the repository is made
available over the network and only the server is directly accessing
the repository, it is still a repository and the server is not
accessing it over the network, so the repository itself doesn't have
anything to do with the network ;)

Yes, unless you have malicious users, in which case you will need the
server. I'm not saying it's not a potential security problem. Just
that co-operative locking of files can work depending on the
circumstances.

:)

--
Michael Wood <esio...@gmail.com>

Ahhhhh......

> Then you should see:

All I see in Eclipse is the REPL view. No Java console view, no "background" job, unless I don't know where to look. After a while, I also get a namespace browser, and shortly thereafter the classes show up in "classes". But this takes around five minutes, perhaps I was simply not patient enough. Or perhaps I started the REPL from something else than the project node.

I simply don't put them into the fat jar. Not as convenient as it should be (in particular since I need to do it again every time I make a fat jar), but at least I get the result I want.

That difference is not easy to see from the menu text. Apparently it is important nevertheless.

Some quick tests work as they should, so it seems my build is correct. Thanks a lot for your help!

Konrad.

How exactly is this possible? Databases *are* servers. "Database" and
"DBMS" are used more-or-less synonymously (when "database" isn't used
more broadly than ACID/SQL/etc.) and the "S" in "DBMS" stands for
"server". SQL is to databases/clients as HTTP GET/POST syntax is to
web servers/browsers. Etc.

I've seen it spelled out as "database management server" innumerable times.

I didn't say "HTTP", I said "HTTP GET/POST syntax". That's carried
over HTTP the way SQL is carried over a DBMS wire protocol.

This is getting silly. "Repository" is a word that brings immediately
to mind typing checkin and checkout commands at a command prompt in
order to work on source code that is stored remotely. And remotely
implies "server".

Thanks. I had the same issue using some generated jar files and this
worked. Here are the concrete commands I used to install the jars into
m2. Posting it here in case someone else finds it useful.

mvn install:install-file \
-DgroupId=com.sforce.soap \
-DartifactId=partner \
-Dversion=22 \
-Dfile=partner.jar \
-Dpackaging=jar \
-DgeneratePom=true

Because in that article, you were (unusual for you) way off base.

I explained it in the part you snipped, but I'll repeat it for you: it
uses an embedded database. A database is *not* a server, it's an
API. The API may well talk to a server - and most do - but there's no
requirement that it do so. This particular embedded database is almost
certainly the most popular SQL database in the world, with hundreds of
millions of installations running worldwide.

SQL doesn't have a position anything like HTTP GET/POST syntax. HTTP's
GET/POST syntax is a wire protocol. Once you it, you can write code to
talk to a server, and get a response. If you know SQL, you can't use a
database *at all*. You have to know an API for the database in
question - and if it's a server, have an implementation that talks
it's wire protocol. ODBC provides a standard API, but not a wire
protocol. Last time I looked, there wasn't a standard wire protocol
for SQL servers.

<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

I was with you until you said "stored remotely". These days, I almost
*never* do checkins or checkouts to a remote server - and I use four
different source control systems on a regular basis: (hg, git, fossil
& perforce). The first three *cannot* do remote checkins or checkouts
- they require the repository be stored on the local disk. The last
one is the only one that behaves the way you describe - and it's a
legacy system I'm trying to get rid of.

As you say, it would be unusual for me if that were the case -- good
reason to suspect that perhaps I wasn't, but rather was
misinterpreted, or something.

Sure it does. Queries are bundled into a wire protocol and sent over
the network in pretty much every system I've seen that used a
database. You have the web server over *here*, and it is visible to
the internet because the gateway forwards port 80 to it *here*; it
talks over the LAN to the DBMS over *here* to run various queries and
build a page to serve ...

HTTP is a wire protocol. The syntax of GET/POST requests is carried on
that protocol (as are HTML files and other files sent back to the
client) but is not the protocol itself.

Not if all you know is "GET /index.html". You need to speak HTTP and
then send that.

Just as you need to know HTTP and not just "GET /index.html". The
difference here is that the databases haven't all settled on one
embedding, analogous to HTTP, for the SQL requests, but all the
vendors use their own different one.

Which doesn't change much of anything. There's HTTP 1.0 and HTTP 1.1
for web servers for that matter, though mostly you can ignore the
differences and, indeed, these days HTTP 1.0 as a whole. It's not hard
to imagine a world where a non-HTTP protocol became a rival to HTTP,
either -- for instance, one with built-in support for mirroring of
static content instead of having to fudge around with things like
Akamai for that, perhaps even one where individual browsers made parts
of their caches (not https stuff, though, obviously, or form contents)
available automatically as alternative sources to take some of the
load off central servers and possibly even maintain some availability
when they were down. If that were to happen, would the things that
interpret GET requests suddenly stop qualifying for being considered
to be "servers"?

Well, the source code is being worked on collaboratively by
geographically separated people in many cases, and from multiple
office cubicle computers in the most geographically-concentrated case.
The canonical master version of the source code must reside in some
single place, which can be at most one of those developers' computers;
so for the rest of them, it's stored remotely.

What's stored locally may range from a single file being worked on at
a time to a copy of the whole code base, but there is still generally
a master copy and there is still therefore some mechanism for updating
the master copy and resolving edit conflicts that collide there.

That mechanism requires the computer holding the non-master copy or
single file to push its diffs to the computer holding the master copy
(and then the latter is a server) or the computer holding the master
copy to pull diffs from the others (and then all the REST are
servers!).

So not only is there a "stored remotely" in there out of necessity for
anything but a single-developer-on-a-single-machine project, but
there's also a "server" in there, or even multiple servers. The
alternatives I can think of are:

1. One developer, one computer. Version control may be overkill in
such a case anyway and it's not how big, major projects are developed.

2. Many developers, one computer. No "remote storage" and if the
developers are co-located no server; otherwise a terminal server. The
former is obviously not parallelizable (though edit conflicts are thus
a non-issue -- single global lock FTW!!!1!1) and the latter is a
throwback to the 1980s or earlier. :)

3. Many computers, one developer manually synching files or just
carrying the codebase around on a thumbdrive. No servers, no remote
storage that isn't simply overwritten when returned-to. The extra
copies, if any, merely amount to backups. Most likely with a
one-developer project with a tower and a laptop, or developing a phone
app on a computer and on their phone.

4. An ad hoc, peer-to-peer system with many evolving versions of the
codebase and patches swapped back and forth but no canonical "master"
copy. This *might* be workable on a small project (a handful of
developers, not too many LOC) but surely cannot scale much above that
without becoming total chaos. There might be no "server" beyond email
in such a case, used for exchanging diff files or whatever. But I
expect any project organized that way to melt down above a fairly
small size of codebase and/or developer headcount. Versions will
become too divergent, the bigger and more numerous they are, until
patches that worked on the sender's copy often won't work, or won't
work without extensive modification, on the recipient's, and then the
ability to share improvements begins to break down rapidly when that
point is reached. In effect, the codebases begin to evolve into
separate species that can no longer interbreed. Perhaps this is how
bacteria, despite being able to share genes in horizontal transfer and
acquire them after birth, nonetheless have distinct species -- they
become incompatible for all but certain broad classes of "plugin
interface implementation" patches such as, unfortunately,
antibiotic-resistance genes.

This could be made slightly more scalable by modularizing, specifying
module interfaces, and letting the modules evolve separately,
versioning each one by patch level, so the version of a module is
bumped every time it's patched. Patches expected to be acquired and
applied in order to keep each module up to date with everyone else's
work. Obvious problem with patch-numbering collisions, where two
developers hack on module Y and both produce distinct patch 1337s
unaware of what the other is doing -- in other words, edit conflicts.
Repositories that number every commit sort-of implement this, but with
a master copy and a database of some sort tracking the changes and the
commit numbers and some mechanism for resolving edit collisions.
Collisions are also detected right away, since both developers will
submit their changes to the central repository. In the peer-to-peer
model each might deliver their own "patch 1337" to a bunch of others
and both could spread for a while among different subsets of the
developers before eventually colliding in one developer's copy of
module Y who receives one of the patches and then, later, the other.
Before that, problems could arise if developer X talks to developer Z
and gets patch 1338 for module Y, tries to apply it, and despite
having patch 1337 it doesn't work because patch 1338 depends on one
patch 1337 and developer X has the *other* patch 1337. Z tells him he
mustn't have patch 1337, he insists he does, confusion ensues until at
some point someone thinks to compare Xs and Zs patch 1337s and
discovers that they're not the same...whence, collision fun again.

So, unless 4 can be made workable, then the typical use scenario for a
version control system involves both a) "remote storage" of code and
b) a "server". Their exact roles may vary. The exact way that
checkouts and commits work may vary. But without those two things in
*some* way shape or form the poorly-scaling anarchy of case 4 above
seems to me to be inevitable. Hell, even time sharing an old Unix
terminal server with a shell-and-emacs account for each developer
means remote code storage (on the Unix box) and a server (if only a
telnet/ssh server). :)

This may be the case, but a "repository" in no way implies that there
are multiple developers involved. Just as (I thought we had agreed
before this thread was revived) a "repository" in no way implies a
network.

Also, even if there is some central, authoritative repository, the
individual developers may (e.g. in the case of Git or Mercurial) still
have local repositories that are not inextricably linked to the
central repository.

The conflict resolution in CVS or Subversion or Mercurial or Git all
happens locally. Not in some central server or repository. Some
crude form of conflict detection might happen in the central
repository, but so what? That does not mean that "repository =>
network", which was your original argument, and it does not mean that
a repository is inherently shared between multiple developers.

If there is a central repository then yes, changes need to be pushed
to it or pulled into it from elsewhere. But the existence of a
networked repository or a repository shared on a multi-user machine
with no networking involved does not preclude the existence of
non-networked, single-user repositories. So again, a repository does
not imply that there is a network involved.

I'd disagree that version control is overkill, but that has is
irrelevant. Anyway, we're not talking about "big, major projects".
We're talking about the meaning of the word "repository".

I only skimmed the last bit. Sorry :) I don't think it changes the
fact that "repository" does not imply a network. All the talk about
best practices etc. are irrelevant to the argument.

--
Michael Wood <esio...@gmail.com>

So, unless 4 can be made workable, ...