So we at GateHouse Media are quickly pushing developers within the organization to learn and master Drupal for many of our smaller, one-off projects. While chatting with Brandon Belew a few minutes ago, it became apparent that manually downloading and installing Drupal and many contributed modules can be quite cumbersome. After reading the Pro Drupal Development book, and becoming a recent Subversion junkie, I've learned to do all of my Drupal and Contributed Module management through a combination of CVS and Subversion.
Through this post, I'm going to attempt to take you through all of the steps of managing your Drupal installation as well as any contributed modules you may need to utilize.
So the first steps are to have CVS and SVN running on either your server or production computer. In my case, I've got them running on both. If you've got a shared host, there are a few tutorials out there on how to setup SVN (many hosts already have CVS installed). Here's a rediculously quick tutorial on how to setup a base install of SVN on your shared server:
wget http://subversion.tigris.org/downloads/subversion-1.4.5.tar.gzwget http://subversion.tigris.org/downloads/subversion-deps-1.4.5.tar.gztar -xzvf subversion-1.4.5.tar.gztar -xzvf subversion-deps-1.4.5.tar.gz./configure --prefix=/path/to/installationmakemake installPATH=~/path/to/installation/bin<br />export PATHFor an in-depth explanation of what these folders mean, see Wikipedia's decent overview.
Now, you'll checkout Drupal into the trunk folder. To do that, run the following command from the parent folder housing those three new directories:
cvs -d:pserver:anonymous:anonymous@cvs.drupal.org:/cvs/drupal checkout -d ./trunk -r DRUPAL-5 drupal
The above code assumes you're looking for the latest stable Drupal 5 version, and that you want to check it out into './trunk'.
After that finishes, you've got a working copy of a Drupal installation straight from Drupal.org. Now we need to get your installation into your own Subversion repository for your personal safe-keeping. First you'll need to create a Subversion repository.
Think of your 'repository' as a bank. It's where you'll put your revision-controlled files, but not where you'll work on them. Once you've created the repository, you checkout versions of your projects and work on them in a remote location (remote as in another directory, or even another computer). The checked out version of your project from the repository is called a 'working copy'. You can then make changes incrementally to the version in the repository with the changes from your working copy.
Run the following command to create a repository named 'repos' in your current directory:
svnadmin create repos
If this doesn't work, be sure that the 'bin' folder from your SVN installation is correctly setup in your PATH. Run the command 'env' to check to see if that path is in there.
Then, go back to your Drupal checkout. At the top directory (where you see the folder that houses 'tags', 'trunk' and 'branches'), run the following command to put the entire codeset (including directories 'tags', 'trunk' and 'branches') into your repository you just created:
svn import name-of-folder file:///path/to/repos -m 'First Import'
There! You've just imported your base Drupal CVS checkout to your repository. At this point, if you wanted, you could delete the source files that you just used to import the code into the repository. I'd hold off until you do the next step.
Now that your project is in your Subversion repository, you need some way to manage and modify that code. The code that you used to import into the repository is in no way connected to the repository. What you need to do now, is do exactly what you did above with the Drupal installation, checkout your files into a working copy.
Navigate to the directory where you'll want your checkout to live, and run the following command:
svn checkout file:///path/to/repos/name-of-project/trunk name-of-destination-directory/
It's important to remember to checkout the '/trunk' of the project, because you don't really need the others, unless you have branched or tagged versions you're working on. Again, see that Wikipedia article for the usage of those folders.
Now you've got a working copy of your Drupal installation from your personal Subversion repository, which also maintains the integrity of your Drupal CVS checkout from Drupal.org. Let's say this checkout acts as the live codeset from which your site is running on. When Drupal.org announces an update, all you need to do is go back to your checked out version of Drupal, and run the following command to update your Drupal codebase with the latest version (assuming here that latest tagged release is '5.7'):
cvs update -r DRUPAL-5-7 -dP
It's as simple as that for upgrading your Drupal installation. Well, not really, if you have modules installed, you may need to perform some additional steps, but for the most part, it's as simple as that.
So now that you've got a nicely updated (from 3 minutes ago) installation of Drupal, you can start breaking stuff (or building, whatever you like). Let's say you find a nice little module on Drupal.org and you'd like to download and install that module into your Drupal installation. With this nice little setup we've got going on, there's no need to download the files manually, dump them into your Drupal installation and FTP to your server. Instead, you can do a CVS checkout of that specific module, then commit to your personal Subversion repository and check out a working copy on your server.
Navigate to:
your-drupal-installation/sites/all/modules/
You may need to create the 'modules' directory if this is a fresh Drupal install. Once you're there, run the following command to grab the 'Image' module from Drupal.org:
cvs -d:pserver:anonymous:anonymous@cvs.drupal.org:/cvs/drupal-contrib checkout -d image -r DRUPAL-5--1-6 contributions/modules/image
Note that the revision tag (DRUPAL-5--1-6) may be different for each module. You'll want to make sure you grab the latest stable revision, rather than the HEAD, as many times the HEAD will contain development (and thus, unstable) code.
So once you run the command above, you'll have checked out the image module, and should now see it in your /sites/all/modules directory. The update process for modules is the same as your Drupal checkout. Navigate to the /sites/all/modules/image directory, and run the following command:
cvs update -dP
CVS will check your working copy against the revision you checked out of Drupal.org for changes. If there are changes, it'll bring your code up to date.
So now you have a Drupal installation and a module as working copies from Drupal.org. You'll want to commit your addition of the image module to your personal Subversion repository for safe-keeping. Run the following code at the root of your Drupal installation:
svn status
SVN will iterate through your installation and notify you of changes that you've made to your local working copy. You should see all of the additional files your Image module CVS checkout created. Run the following command to 'commit' (save) these changes to your repository:
svn commit -m 'adding cvs checkout of image module from drupal.org'
Then, you can run a status again:
svn status
There should be no more pending actions. When you run 'svn status' and it returns a prompt with no actions, that means there is no difference between your LOCAL changes and the repository version. However, it does not necessarily mean there have not been changes to the repository. After all, if someone applies a patch to a piece of code in the repository, it does not automatically update your working copy for obvious reasons. You can see if there have been changes to the repository by the following command:
svn status --show-updates
So now, you've got a Drupal installation with your custom or contributed modules all under a personal Subversion repository, as well as copies of your Drupal installation and modules tied to a CVS checkout for upgrading (and downgrading) with ease.
Oh, here's a few links I was attached to when I first began this debacle:
At this point, I'm pretty much out of breath and I think we've covered a fair amount of ground.
So, yeah, well, I'm done.
31 Comments
I'm not AS low tech as I used to be ;) I do use SVN occasionally.
I'm probably going to sync my drupal build tonight.
Hey man, thanks for this. This is almost exactly the same process I use daily and was wanting to create an article on this particular subject. Looks like you beat me to it ;) I'll put up a link to this, but consider replicating it on the d.o site for others, just in case yours goes away for some reason in the future. Nice job!
This is great. I am doing this on several sites and agree that it is a fantastic all around solution.
Now every once in a while I run into the following problem, and wonder how you handle it.
1. A module is modified in Drupal CVS contrib repository, one existing file is deleted and two files added.
2. Upon doing a CVS "cvs update -dP" the CVS client brings me up to date, deleting the old file and downloading the new ones.
3. When I do the SVN status command, on the one hand it advises me about the new added files; but is completely confused ("!") by the "missing" file.
What do you do to handle this gracefully?
Thanks,
Victor Kane
Victor,
I think I understand your problem. What you'll need to do is just run:
svn delete 'name of file marked with !'
SVN is looking for that file in the directory, but since you never told it that you were deleting it (because CVS did it for you), it is assuming it should still be there.
Essentially, when you delete a file from a directory under version control, you should always use 'svn delete' rather than 'rm' to make sure the changes are reflected in your SVN repository (unless of course you're only temporarily removing a file for some reason or another).
After you run the 'svn delete' command, your 'svn status' should show that file previously marked with '!' now marked as 'D' ready for deletion from the repo.
Nick,
that's not a graceful solution, which is what Victor asked about. I'm pretty sure that what you described, is what he's doing now.
But I'm afraid there *isn't* a much more graceful solution, except for writing a script that does this automatically.
I have a more scalable solution than what you've described here. "More scalable" in the sense that you don't have to repeat the same task for every site (e.g. update a module). I will publish the article on my site when it's ready.
Ah, missed the 'graceful' part and just went for the 'working' part. Looking forward to your solution!
[...] Nick Sergeant :: Blog - Painless Drupal revision control with CVS and Subversion on a shared host. I probably need to look into something like this soon. (tags: drupal webdev howto) [...]
Thanks for this tutorial. I'm a newbie to the CVS/SVN stuff, so this was helpful. I'm still trying to wrap my head around some basic concepts. Perhaps you can help... Hopefully my questions make sense...
1. You wrote that "The code that you used to import into the [svn] repository is in no way connected to the repository." Does this mean that when I update my CVS copy of Drupal the changes are not passed to my SVN repository? If so, I'm not sure how this helps me keep my Drupal sites up-to-date. I was thinking that the CVS update should pass to my local SVN repository to keep my local Drupal installs up-to-date. Is this possible?
In other words, how do you keep the SVN Drupal & Contribes current?
2. Why do you need to use CVS and SVN? Couldn't you just use CVS and checkout various working copies of Drupal to use with each site you are developing? Wouldn't this apply the updates from Drupal.org automatically?
Thanks for the guidance and clarifications.
Jason,
When I mentioned that your original code is not in anyway connected to the SVN repository, I meant that you'll need to checkout the project from the repository after you've imported it. Think of importing code into Subversion as a one-way street. If you want to modify or 'enable' revision control on your code with the Subversion repository, you would replace your original code with a checked out working copy from the repository, and use that instead of the original code you used to import into Subversion.
Once you're working with a checked out working copy, you're on a two-way street, where you can make changes to the SVN repository version, as well as update your checked out code with any changes from the SVN repository.
When you're working with a checked out working copy, you can run CVS updates that will modify your checked out copy. However, those changes from CVS will not be made to your SVN repository until you tell it to do so. If you ran a CVS update that created a directory '/new', you would then need to run 'svn add new/' to tell SVN that you'd like the CVS changes to also be reflected in your SVN repository (not just your working copy). Once you do that, and then run 'svn commit -m "explanation of changes"', the changes will be saved to your SVN repository.
To answer your second question, you most certainly could just use Drupal's CVS to manage your installation as well as any modules from Drupal.org. However, this removes the benefit of having your own version controlled codesets.
For example, if I checked out a base install of Drupal, as well as the 'Image' module from Drupal.org using only CVS, I would have a perfectly working setup under CVS revision control. However, if I decided I wanted to add a custom theme at '/sites/all/themes/custom/nick-theme', that would not be under revision control, because it doesn't live (and is not checked out from) Drupal.org. Therefore, we use Subversion to add the version-control benefits to our local, personal combinations of Drupal and Drupal contributed modules, without removing the CVS connection to them on Drupal.org.
Nick, Thanks for getting back to me. I think this is clarifying things for me. Please check my thoughts and see if they are correct now:
1. I can still use CVS to update my working copy that is under Subversion control. To keep up with the new Drupal updates I use a CVS update on my SVN working copy. I don't do any updating to the ORIGINAL CVS checkout I got from Drupal.
Creating a SVN repository from the CVS checkout still gives me CVS functionality within the Subversion working copy. Is this correct?
Is that the purpose of all the CVS files inside my SVN working copy?
Alternately, I could have simply created a SVN repository from a downloaded Drupal copy, but this would not give me any updates directly from Drupal. Right?
2. The CVS only scenario doesn't work so well because you cannot make changes to the repository since it exists at Drupal.org.
I tried to create the /modules and /themes directories in my CVS checkout, but they didn't get passed to any working copies.
This is because they would need to be committed to the repository before they show up on new working copies. Correct?
Therefore, SVN is a good way to go...
Thanks for your time and patience.
Jason,
Correct on both counts. You can essentially delete your 'original' CVS checkout of Drupal from Drupal.org so long as you've commited the project into Subversion. At that point, you could checkout a working copy and work with that.
Also, you are correct in assuming that importing your CVS checkout of Drupal will retain your CVS functionality within your Subversion working copy.
Excellent help Nick!
One more question on best practice with SVN:
Let's say I want to create two or more Drupal websites on my local machine using SVN. I almost always have a few Contrib modules that I want on each site (Views, CCK, Admin Menu, etc). I'm assuming I'll add these to my SVN repository via CVS like you explained originally.
Now after that, if I want to customize each Drupal site do I create a new branch in the SVN repository for each website I'm customizing?
In other words, what is the best practice for using the SVN repository to create various Drupal sites (branching? Using more than one repository? etc)?
I promise to stop bugging you about all this soon!
Happy 2008!
Jason
Jason,
No need to stop bugging - it's how we all learn :)
Good questions. You most certainly could do multiple checkouts of one SVN project for each of your sites. However, creating branches for each of your individual sites might not be the best solution. I haven't run into this yet, so I'd love to hear from others who might have been down this road.
If I did need to do this, though, I think I would write a simple bash script to quickly do a CVS checkout of Drupal and the common contrib modules I'd like in all of the installs, and then import the checked out code into SVN as a separate project.
I think you might find that easier to manage than having multiple branches for each of your sites.
On a separate note, you might be interested to checkout Drupal's built-in multi-site functionality. Drupal actually makes it rediculously easy to have several sites running on one Drupal installation (and thus, one codebase / one SVN project). It allows for individual configurations per-site on just one Drupal installation, too.
Thanks for the advice. I'm too new to SVN to know whether creating a branch per site is a bad idea. I can't use the multi-site option with my development because eventually each site will stand alone on a different server.
Looks like it is time to learn how to write a bash script...
We never stop learning!
Thanks,
Jason
Ahh - I see. Happy coding!
[...] Drupal from Drupal.org onto your server. About a month ago, I wrote a blog post outlining steps to manage your Drupal installation with CVS and Subversion if you’re feeling [...]
Hi,
This is an excellent tutorial. Thank you so much for sharing it. I'm wondering if you could elaborate a little as to how you are using this method to deploy sites?
Specifically, I know many folks use a dev/staging/production set up. Is this what you do? If not, what are you doing to deploy and make changes to your production site using the cvs/svn setup?
Any help is greatly appreciated.
Thanks,
Mike
Mike,
Thanks for the comment.
Excellent questions as well. I'm contemplating writing up a detailed explanation of how we manage dev vs. prod installations for multi-site Drupal projects.
It essentially involves having a separate branch for 'dev' code which is eventually merged into the trunk and stable releases (tags).
We've been through a few development scenarios, but the above appears to work best, and is generally the process adopted by many open-source projects.
Of course, managing dev vs. prod installations of Drupal involves some MySQL fancy work, which is why this topic may deserve its own blog post :). Stay tuned.
"Of course, managing dev vs. prod installations of Drupal involves some MySQL fancy work, which is why this topic may deserve its own blog post :). Stay tuned."
Indeed! My head is SPINNING on this topic. With the database-centric (or should I say heavily database dependent) configuration of Drupal sites, I am finding getting my head around version management in the context of deployment a real challenge. I still haven't worked it out and right now, due to the wide range of recommendations out there, I'm just going with the trial and error method. Not my preferred method.
Thanks! I'm truly looking forward to your post.
[...] I do is either preceded or followed by some sort of Subversion interaction. I wrote about my Subversion obsession a few months ago. So normally I go about my business happily, whistling away while I commit changes [...]
Nice article.
BTW, nice to see that you submitted announcement at drupal.org
hey nick,
came across this article by accident but now that i have, i'm really interested in your followup that would document test and prod parallel sites implemented in svn.
are you still planning on doing that and if so any ETA?
thanks bud
I plan on doing alot of things, but I've been stupid busy the past few weeks. I still need to finish fixing some broken links around the site, in addition to actually getting a taste of my own medicine and switching this very site over to Drupal ;)
That said, I am still planning on doing a Drupal dev/stage/prod post, so stay tuned!
[...] some Googling I found an article by Nick Sergeant and he basically had the same routine. But he dumped 1&1. I can't afford to [...]
hey nick,
came across this article by accident but now that i have, i’m really interested in your followup that would document test and prod parallel sites implemented in svn.
are you still planning on doing that and if so any ETA?
thanks bud
resimler,
I'm going to try and get that blog post together sometime next week. We've been employing a fairly sturdy SVN dev -> prod setup with one of our latest launches, so I've got some good ideas on best practices. Stay tuned!
Nick
Nick, Thanks for this, very helpful.
Excellent article, Nick.
I have a problem and hope you could give some suggestion on a solution.
We have a structure of two working copies for each of the developer and then a "dev/stage" copy which is auto updated using post-commit hook. This system works fine as long as apache doesnt try to create a file. But while using imagecache module to dynamically create the images we ran across permission issues where Apache wasnt able to create the image because it didnt own the files/imagecache folder. Hence to have a work around for this we had to write a script where the imagecache folder's owner was changed to "user" before commit and "apache" after commit.
Is there any other solution for this problem ??
Thanks for your comment. That's an interesting scenario you've got there. Would it be acceptable to set the imagecache folder to 777? Is that folder only responsible for holding temp files while generating the dynamic images? As mentioned, I'm not familiar with the imagecache module, so I'm not sure exactly how it works.
Nick
I have one comment regarding the use of CVS checkouts for modules: you loose the extra lines in the .info files added by the project module at drupal.org:
version = "6.x-1.0"
core = "6.x"
project = "advanced_help"
datestamp = "1210219202"
These values are not available in the CVS versions.
For me, this is a showstopper, because I really want the update detection from Drupal 6, which requires these values.
Jorrit,
The CVS Deploy module should take care of that for you. Hopefully we'll see better support for this in D7.
Nick
Post new comment