Managed Code Deployment
PhreaK
Posts: 966
So here's the situation, we have a large-ish number of deployed masters (150-ish) which are worked on by a few of us tech heads here at JAG. Currently all of our operational code is managed in an SVN repo with the trunk containing the currently deployed system code and new features developed in branches before being merged, compiled and manually deployed. I'm currently in the process of migrating us across to Git (because its awesome - if you're using SVN I highly recommend you consider making the switch) and am looking at ways the current workflow can be improved to make things a bit more efficient and points of possible human failure removed.
My main fear with the current setup is that our code repository is completely decoupled from the operational code. Currently there is nothing to stop someone from making a modification (bug fix etc) and committing it back to the repo without it being dumped to the target master, or even worse, making a change, dumping it to the master and not committing back to the repo.
What I'm looking at doing is switching the workflow to have our code management server handle all the deployment as well. This would involve setting up a hook script so that after anything is pushed back into the master branch on the repo server it is compiled and deployed from there (during a specified maintenance window). Tech's could still bypass this say for example if an urgent bug fix was required by pushing straight to the masters but a checksum could be used to compare the code on the repo server and what's running on the AMX box and if there's a conflict shoot us an email so that we can then resolve it.
My question is has anyone else implemented something similar in the past? If so what has your experience been?
My main fear with the current setup is that our code repository is completely decoupled from the operational code. Currently there is nothing to stop someone from making a modification (bug fix etc) and committing it back to the repo without it being dumped to the target master, or even worse, making a change, dumping it to the master and not committing back to the repo.
What I'm looking at doing is switching the workflow to have our code management server handle all the deployment as well. This would involve setting up a hook script so that after anything is pushed back into the master branch on the repo server it is compiled and deployed from there (during a specified maintenance window). Tech's could still bypass this say for example if an urgent bug fix was required by pushing straight to the masters but a checksum could be used to compare the code on the repo server and what's running on the AMX box and if there's a conflict shoot us an email so that we can then resolve it.
My question is has anyone else implemented something similar in the past? If so what has your experience been?
0
Comments
I'm doing something similar here down at Swinburne Uni where we have 300+ masters and growing. I've used subversion to manage my code, but the hard part has been ensuring that the most recent bug fixes are deployed to each room. I'm building a centralised server-based code deployment system using PHP/MySQL and the well-known FTP tricks for non-NLS code updates. The target audience is large organizations who have the same code in multiple rooms. (Probably not useful for the residential market, but useful for us in education/law.) The system supports multiple room types.
There are various roles in the deployment system: Programmer, Change Control Manager, etc. Some have authority to upload code changes (but not deploy), while others (not necessarily programmers) have authority to push code out to systems. The idea is to provide controlled deployment to prevent a rogue programmer (not that we have any!) from deploying bugs far and wide, and allowing managers/techs to commission "cookie cutter" rooms.
A programmer works on a project using subversion to track code changes. Once they are satisfied that it is stable, they can then upload it into the deployment system, where it can be pushed out on-mass to multiple masters. I've also built a module which periodically checks in with the deployment manager for "new code for room type X". When new code is available it is deployed. The plan is to eventually "window" the availability. E.g. 2am - 5am, && system not running. Rolling out a bug-fix to 70+ rooms during business hours is not fun. Staying up late overnight or weekends is also not fun.
Another "nice" feature is that it will allow me to push a program out to a master based purely on the masters IP address. When a contractor installs an unprogrammed DVX-2100 into a classroom and provides me the IP address (via the DVX LCD) I (or a non-AMX-programmer with suitable deployment credentials, like mymanager) can push the code to it from the server. It also sets the Duet memory allocation, sets the clock, etc. It should be able to set the hostname and anything you normally do from a telnet session. I'm still working on remotely setting the security - figuring out http cookies from PHP...
I've been planning on dropping you a line about this, since I think you may find this useful. Once it has reached maturity the plan is to open source it. Give me a call if you want to chat further about it. I'd like to collaborate if you'd find it useful.
Yours,
Roger McLean
Swinburne University