Using Git with Salesforce and distributed teams

Introduction

I’ve been asked several times (and have presented a couple of times) on how our team handles doing Continuous Integration (CI) and Continuous Deployment (CD) with a distributed team. However one of the points that I’m always asked is specifically how we use Software Configuration Management (SCM) with this processes.  For this blog post I’m going to do a deep dive into our process.

NOTE: This process is by no means the end-all-be-all process for everyone.  This is just what we have come up with (generalized for public consumption) over several years of developing on the Force.com platform

Prerequisites

Git

Get yourself a git repository. This can be a self-hosted git repository or it can be one of any number of cloud-based hosting 1 2 3.  You’ll need this because this is where ALL of your code an configuration should live. For the most part, if it doesn’t exist in git, it doesn’t exist. In other words, don’t expect something that is not in git to stick around for a while.

Now git is not the only SCM out there (I don’t want to start an internet argument over if it’s the best/worst/etc) but it’s what we are using and it’s what all my examples will be in. If you are comfortable with another SCM the process should be fairly easy to adapt.

Development sandboxes

You should have a communal development sandbox that all of your testing and verification of new features occurs.  This should be a full sandbox with all the data from your production org.  If you can’t get a full sandbox, this should at least contain a sampling of production like data.

Every one of your developers should have their own developer sandboxes. These should just be “developer” and I would recommend you have some test data that you put in there.

A way for developers to pull updates from the sandbox

There are lots of different ways to do this, but the biggest thing is the data that comes back needs to be consistent. We use Solenopsis, but you can use Mavensmate, the Force.com IDE or even roll your own Ant tool.  Not every developer has to use the same tool, but it’s important that you are able to do a destructive change push that overwrites both objects and code.

Automated build process

While this is not required it is highly suggested. Not only will this ensure that your development environment is always updated, and will save some poor sap from having to do this manually. We use a combination of Jenkins and Solenopsis to do all of our pushes from CI development instance all the way to our release to production.  Once you get it into your development sandbox, however you want to get it to production is up to you.

Workflow

This is where the day to day workflow is done.  The following shows the life-cycle of a feature being developed on the platform.  For the examples I will be including the Solenopsis commands. Where this command is, feel free to replace with whatever the equivalent is with your build application.

Push to sandbox

Every feature should start with a clean slate. You should pull from git to make sure you have the most recent feature set and then push all the changes to your developer sandbox

git pull
solenopsis destructive-push

Create a feature branch

In order to make sure that all work is cordoned off and multiple commits can be made without dirtying up the master git log

git checkout -b myfeaturebranch

Do config work

While this can be done at any point in the feature development, I prefer to do this first.  This way when you pull down your configuration work you do not have to worry about other things accidentally being overwritten.

After the configuration work is done, pull the changes down to your local machine and commit them. I like to commit them after pulling them down because that way I have a cleaner repo listing for when I use Solenopsis’ git-push command.

solenopsis pull-full-to-master
git add objects/MyObject__c.object
git add objectTranslations/MyObject__c-en_US.objectTranslation
git checkout .
git commit -a -m "My configuration changes"

NOTE: While not necessary, I prefer to use git add to add the individual files I changed and then use git checkout to remove the others.  The keeps me from accidentally making unwanted changes.

Do development work

If you’re a developer, this is where you will spend most of your time (surprise!).  After you have your code where you like it and all your tests are written (you did write tests didn’t you?) then commit that code to git.  The steps below can be done as many times as you see fit. I often will do a commit and change things and then checkout my old commit if I don’t like where the new code is going.

git add classes/MyNewClass.cls
git commit -a -m "My coding work"

NOTE: Any new files that you create will have to be specifically added with a git add, files that are already in git but just modified will be added with the -a flag for the git commit.

Rebase

If there is a tricky bit in this whole process, rebasing your code is it.  Depending on the size of your team, the length of your feature or the timing with the rest of your team, odds are pretty good that someone else will have finished their feature and pushed it to git before you’ve finished yours.  Now before you are ready to push your code out, you need to rebase with master.

Rebasing will take the commits you have done during the feature development and re-play them over the top of the code that exists in git.  Most of the time this will go great (assuming you are working on new code or parts of the code that nobody else is working on), however, sometimes you’ll get the dreaded merge-conflict.  I have found that the best way to handle merge conflicts is to prevent them whenever possible.  Here are some of my tips on how to prevent merge conflicts with apex code

  • Whenever possible, coordinate with other team members about which classes you will be modifying.  If you can, try to stay away from the same code someone else is working on
  • Add new methods to the bottom of your class file.  This won’t necessarily mean you won’t get a conflict, but it will make cleaning the conflict much easier
  • Have a good understanding of what the code you are modifying does. If you don’t understand how the code you are modifying works then it makes doing merges much harder to tease apart.
git checkout master
git pull
git checkout myfeaturebranch
git rebase master

The git prompts are pretty straight-forward when it comes to dealing with conflicts, and there is some pretty good reading on how to fix it.  What I do is look for the “>>>” string in the code that the conflict occurs and then combine the code myself.  Most of the time it’s as simple as just removing the markers and occationally adding a new closing brace.  Sometimes it’s much more time consuming.

Rebase early and rebase often

If you can take away one lesson from this, it’s “rebase early and rebase often.”  There is zero harm in rebasing everytime someone else pushes a commit to master.  To be honest, it will probably make your life much easier.  The more work you have done and the more that has been committed to master, the more you’ll have to deal with on a rebase.

Push the rebase

This step is only required if there was something to rebase.  If you got lucky and there have been no changes to master since you first branched, you can go ahead and skip to the next step.  Otherwise, go ahead and do a full push to your sandbox and then re-run all your tests, making sure that you didn’t break someone else’s tests and they didn’t break yours.

Commit to master

This is the moment you’ve been waiting for.  You’ve got your feature complete, you’ve rebased and all your tests pass.  Let’s merge! Now before you get all silly and just do a plain merge, ask yourself some questions:

  • “Self, have I made all of my commit messages cleanly and not something like ‘adding code'”
  • “Self, do I only have one commit?”
  • “Self, do I like a sloppy commit log?”
  • “Self, do I want to cherry-pick out 20 commits to roll this feature back if I broke something?”

If you answered “no” to any of these questions then guess what! squashing the commit is for you!  Squash committing takes all of your commits and condenses them down into a single commit.  Not only does this look better in the git log, but it also makes it much easier to tease apart if something goes sideways and you have to roll back a commit, or if you have to cherry-pick a commit to go to production before the entirety master is ready to go.

git checkout master
git merge --squash myfeaturebranch
git commit -m "UID - The awesome new feature"
git push

Take this opportunity to add any sort of tracking information for your feature.  We put the UID of the story in front of the commit message so we can track it to a unit of work

Cleanup

After you’re done with the merge it’s time to clean up that local branch.  No point in keeping it around anymore.  If you need to do more work on it, you should re-branch off of master and start the whole process over again.

git branch -D myfeaturebranch

NOTE: You will have to use the -D flag because we did a –squash merge.  Because of this the history tracking of all your intermediary commits are lost and thus the branch appears to be un-merged.

Automated push

At this point whatever automated system you set up should be monitoring your git repo, see there is a new commit, and do the push.

I would recommend that you also set your automated system to run the tests associated with your organization.  Because our tests can take in excess of 2 hours to run, we disable this run on ever push to the development sandbox.  However, we do have jobs that run twice daily that just run the tests in our development sandbox and notify us if there are failures.

Caveats

Things not stored in git

Unfortunately, not everything can be stored in git.  The things that cannot should be tracked and done manually in your development sandbox.  We make it the responsibility of one person and have it tracked in a document that follows along with the full release process

Config in git

One of the biggest hurdle for any team switching to using git is to also have their configuration in git.  If the configuration changes are not in git then you have the problem of either clobbering configuration changes in your development sandbox, or having all of your users having to make the same configuration changes in their personal sandboxes.  While it will take a bit of work to get any admins that are not very savvy with git up to speed, I’ve found it doesn’t take long and you’ll find that your admins will get the hang of it quickly.  Also, since most configuration changes happen in the XML files there are rarely any merge conflicts that occur.

Somethings cannot be removed with ANT

Unfortunately there are just some things that cannot be removed without manual intervention such as picklist values or tags.  These will have to be done manually to every sandbox as well as to a production org.  To help with this, we keep a document for each release that denotes the manual changes that must occur after the release is done.

This entry was posted in Development, Salesforce and tagged , , . Bookmark the permalink.
  • Bobby

    Great write-up. I’m experimenting with Solenopsis and I’m having trouble doing pushes from one org to another. I’m getting a lot of deployment errors related to non-code metadata (e.g. fields missing from page layout, custom field type-change). How are you dealing with this problem in your process? I could envision how this could work just for code-related metadata changes, but right now the tool doesn’t seem to solve the problem of metadata dependency deletion/deployment.

  • We do almost all* of our changes via Solenopsis including layout and custom fields. What was your procedure to get the data and then deploy to the other org? It’s not great for doing a dramatic change in an org, it’s designed for incremental changes from orgs that have been cloned from each other (ie sandbox cloned from production).

    There are some standard layouts that we had to add to our sfdcignore file (–ignorefile or sf.ignoreFile) that we don’t use and for some reason Salesforce won’t let us push.

    Current Ignore File
    ==========================

    profiles/Customer Portal Manager Standard.profile
    profiles/Gold Partner User.profile
    profiles/Guest License User.profile
    profiles/High Volume Customer Portal User.profile
    profiles/Chatter Free User.profile
    profiles/Chatter Moderator User.profile
    profiles/Chatter External User.profile
    profiles/hubcase Profile.profile
    workflows/Case.workflow
    workflows/Question.workflow
    workflows/Reply.workflow
    workflows/SocialPost.workflow
    workflows/SnapshotAuditEvent.workflow
    workflows/SnapshotBin.workflow
    workflows/SnapshotConfig.workflow
    workflows/SocialPersona.workflow
    workflows/Case.workflow
    workflows/Question.workflow
    workflows/Reply.workflow
    workflows/SocialPost.workflow
    workflows/SnapshotAuditEvent.workflow
    workflows/SnapshotBin.workflow
    workflows/SnapshotConfig.workflow
    workflows/SocialPersona.workflow
    objects/Idea.object
    reportTypes/Ideas.reportType
    layouts/PricebookEntry-Price Book Entry Layout.layout
    workflows/QuickTextAction.workflow
    layouts/FeedItem-Feed Item Layout.layout
    workflows/DuplicateRecordItem.workflow
    workflows/MacroInstruction.workflow
    workflows/FeedItem.workflow
    workflows/StreamingChannel.workflow
    layouts/Macro-Macro Layout.layout
    workflows/MacroAction.workflow
    workflows/Macro.workflow
    layouts/DuplicateRecordSet-Duplicate Record Set Layout.layout
    workflows/DuplicateRecordSet.workflow
    workflows/ExternalEventMapping.workflow
    workflows/UserProvisioningRequest.workflow
    objects/Opportunity.object

    ==============================

    Some of these were added when production was a different API version than our development sandbox and can probably be removed (like Opportunity) but we don’t use them so it’s not been an issue.

    * Somethings just can be done with it like Call Center configuration, etc

  • Bobby

    I see. I pulled changes from “master” and tried both a destructive-push to the “dependent” env and encountered 165 errors from 4587 components. I’m OK with hitting the reset button on our environments and going from there. However, how do you handle non-code metadata changes with git? For example, along with a VF page and an Apex controller created using MavensMate, I created a custom field in Salesforce. Do I need to refresh my MM project from the server and let the custom field change flow down to my local which git will pick up? At that point I commit all the code changes and the associated object metadata in git. Using Solenopsis, could I then do a git-push to get that up to UAT?

  • I can’t speak for MavensMate but when we do any metadata change we do a `solenopsis pull-full-to-master` to pull down the entire metadata structure and then commit any field changes to git.

    `solenopsis git-push` will only push files that are unstaged in git. You’re better off doing a destructive-push or a delta-push to get it to UAT.

  • Bobby

    OK, got it. Speaking of delta-push, I’m getting a runtime exception during a dirDiff operation. It seems that it is trying to find the folder /abels/CustomLabels.labels when doing a diff on that metadata type. Of course /abels/ is not a folder, but /labels/ is. I tried to find a potential typo or issues in the source code, but couldn’t hunt anything down.

  • Weird. Do you mind filing an issue on github with the output from the command-line so we have some context. I couldn’t find any reason with a quick glance.

  • Bobby

    A co-worker through out the possibility of character escaping. Do you think there is anything in that? At the moment we’re roadblocked on using Solenopsis unfortunately. Cheers.

  • I don’t know. If you could file an issue on github with the errors you’re seeing and the output I can try to figure out what’s going on. Comments on the post don’t make it the easiest thing to track and work.

  • Garrett Zaino

    Hey @grepsy:disqus – thanks for taking the time to write this up, great and really helpful.

    We recently built our CI/CD process and found a similar approach to be right for us. I can’t help but feel we missed something somewhere. Whenever we need to rebase our personal environments we are left with having to go through 4000 lines of xml per profile to ensure the proper fields are listed and captured.

    What I’m finding is Git is marking up the xml files in a very complex way. It ends up creating 60 merges when I simply added 1 field to the profile.

    I’d like to make sure I accommodate the “clean with master, I add a field, someone adds a field to master, now I need to resolve the profile to keep both fields” issue. Could you expand on the checkout process for configuration work? It seems like whenever I want to pull from Git I need to go through all of the metadata files I’ve been modifying and make sure they are resolved with Master.

  • I’m not quite sure I understand what your issue is. There should only be a couple of lines of XML in your profile/object files.* If you are talking about new lines being in place when someone else on your team add files and then you rebase, this is normal. If you are having to manually review the rebase and manually merge these XML files, then something is not configured correctly with your version of git. Since XML is highly structured I’ve don’t think I’ve ever had a merge conflict on any of the XML files.

    It sounds like you may not rebasing frequently enough or you are not doing a full push to your sandboxes before doing your changes and re-pulling.

    * This of course can mean hundreds of lines if you have a bunch of profiles…

  • Garrett Zaino

    Thanks, I think that clears up some confusion for me.

    Regarding the “There should only be a couple lines of XML…”, I’ve been under the impression that I need to do a full retrieve of the Profile (i.e. pull all object, visualforce, apex, etc) so the Profile has all the information when I go to commit to Git. It sounds like that’s not the case – I can rebase and that will keep the existing XML and merge in any new XML.

    I’ll try that approach and try rebasing; previously I would retrieve from Dev then pull from Master and create conflicts which I would resolve manually.

  • Cristiano Sinadino

    Hi @patrick Connelly Thank you so much for all your hard work and contributions, we are a stronger community because of people like you. Cheers !

  • Guest

    Hi Patrick – Thanks for this article. I agree with the concepts you described in this post, and am hoping to standardize our source code control and CI at my company.

    There was one thing I’m still seeking clarity on which is, how to properly structure the repository. How do you accommodate a repo that has numerous projects with different release timelines that will be deploying to the same Production org? I’m interested in how to set up common code, and how to properly branch. Would you advise on branching for each project?

    Thanks for your work and contribution!

  • Our team works on individual stories but they get released to our production org in one chunk. If we have a story that is going to take multiple releases to complete then we will get that in a private branch and it is deployed only to the developers sandbox. Then when the feature is complete it is merged into our master branch and that is automatically pushed to our main development sandbox automatically.

    You would probably do the same thing with a branch and sandbox per project. If you have multiple developers working on a single project then you would want to have a way to automatically build to that projects sandbox. Part of your process should also be having someone rebase from the master branch frequently to reduce the chances of a merge conflict. And then whenever it is ready to be released to production, then you would merge it into master, push it to your main development sandbox and do your full suite of testing.

    As for the “on disk” structure of the repository I would recommend having a ‘src’ directory off the root of your repository and all of you Salesforce deployment code lives in there. Then you can create other directories for additional build scripts and the like (see [1]).

    [1] https://github.com/apex-lodash/lo

  • Guest

    Thanks Patrick…I’ll look into this.

  • Enrico Murru

    Awesome article! I’m still struggling with the need for the customer to have only one Full Sandbox with several stories that are deployed in production in different times (and sometimes this leads to corrupted deploys), but your workflow helped me to make some of the steps clearer.

  • If you only have one full sandbox should always mirror whatever you currently have in production. You should use developer or developer pro sandboxes for you multiple stories. Then when you are ready to deploy, deploy it to your sandbox and do your validation testing. Then push from that sandbox to production.

  • Enrico Murru

    The only problem is when the customer wants to test both stories in the same (full) sandbox but wants only one released into production…I agree that the full sandbox should be the mirror of production. Thanks for your response!

  • Manjunath TJ

    Thanks for the detailed flow of Git usage. But am struggling with deployment of specific Salesforce components.

    Say an Object already has 10 Workflows and for release work, modified 2. But certainly we don’t want to deploy all 10 Workflows to next Sandbox. Rather want to do only the 2 modified changes.

    How do we do in Git, since the branch repo is a mear close of master with complete code base.

    Thanks for any inputs in this regard.

  • So you are talking about only deploying a subset of workflows? Like you have 8 workflows from story A and two from story B and you only want to move the ones from story B to your next sandbox?

    If that’s what you want to do, then you’ll want to just cherry pick the commit into a tag or another branch and then deploy that tag / branch to your next sandbox.

  • Niatnuom Drahcir

    Hi Patrick, What’s your strategy for dealing with a full sandbox refresh?

    Is your master branch initially committed from production or your full sandbox?
    If it’s the full sandbox, do you pull down the full set and commit that each time the sandbox is refreshed? Also, do you refresh the developer sandbox at the same time? how do you deal with intermediate developer sanbox refreshes?

    I’m sort of following your approach below, but because I haven’t convinced 2 other developers who work on the same org to use the same approach, i’mm pulling in their changes myself each day and it’s getting a bit boring!!

    Thanks for any pointers!

  • What’s in the full sandbox is typically slightly ahead of production. We release to production every 3 weeks so there is stuff in git that has not made it to production.

    Whenever we refresh any sandbox (be it a full sandbox or a developer sandbox) we simply re-run our deployment to bring the sandbox up to what’s on master.

    The only way for this to work is for no changes to be made directly in your full sandbox and only allow changes to be made in developer sandboxes and pushed to git. Then your deployment process will push it to the full sandbox. It’s a thing that can cause contention among some developers (especially ones that have always done their work like this in Salesforce), but the benefits of not losing work and having everyone on the same page well outweigh the little bit of technical overhead.

  • Hi Patrick, do you have a video of your CI & CD in SFDC presentation?

  • Unfortunately, no. When this was given at Dreamforce, they were not recording developer theater talks.

  • Scott Reser

    Thanks again for this post! Very helpful! We’re trying to get ourselves setup from a version control standpoint as we get used to SF more and more. I’ve seen conflicting information off and on about what can be deployed via Force.com API. I’d been getting confused by what was possible to manage via Eclipse IDE. And it seems like Eclipse can get some of the important bits, but there’s still a lot of the declarative stuff in SF that it can’t pull down. It sounds like ‘everything’ more consistently now and the way to get *all* is via Ant, but while you can /do/ everything there are things you either shouldn’t or maybe wouldn’t want store/manage/deploy in Git. You’ve mentioned that as well here, mostly that not everything is in Git and that there’s still some manual changes that you have to make. I’m wondering if you could share what those metadata types are that you’ve chosen *not* to store in Git, and there by manage deployments of those types via other means. ssosettings was one thing that we have that jumped out at me. Someone else suggested sites and reports. I’m basically looking for a list of metadata types to avoid putting into version control. Thanks!

  • If you can pull it down into git, do it. The things that are not in git are things that can’t be, not things that we don’t want to be in git. Most of our manual changes have to do with things that have to be done in a certain order or are better to do via the UI (like picklist replacements). There are some metadata types that we haven’t started pulling in because they cannot be re-deployed and we have not gotten them 100% working with our current tooling. These would be things like Entitlement Processes. We are working on getting them pulled in, but since they change rarely, it is not a high priority to do so.

  • Scott Reser

    Thanks for the reply! I had a chat earlier with someone from Gearset who provides some interesting deployment management & monitoring tools. For someone like me who’s still trying to understand the options, it’s pretty crazy what they can do. But the chat left me with the impression that there’s things we likely wouldn’t want to track in Git. They floated site.com, I floated SSOsettings. The fact that the Force.com IDE doesn’t seem to support downloading all metadata (from what I’ve seen), supports this to some extent from how I’ve approached it.

    We’ve been leaning toward having separate branches: Master, Staging (full sbx), Dev (dev sbx, and then personal or feature branches from there with corresponding dev sbxs as needed). If I’m understanding it right (and I might not be), the ssosettings (while it could be stored in git, for example) would be different on a dev/staging org compared to production/master. If I’m deploying everything on a given branch to its corresponding org, going back to sso (as an example) I’m thinking I’d end up with production settings applied to a dev/staging org where it wouldn’t work. Or conversely dev/staging settings applied on production.

    I think the message I’m picking up on is that if there’s things that don’t change often, if ever, there’s less of a need to try and track those changes clearly. Ultimately it’d be great/valid to do so, but as is the benefit might be debatable.

    We’ve initially tried to come at this the same way I think your suggesting, store everything in Git and deploy everything from Git. But I’m trying how to resolve what seems like exceptions to the rule.

  • ricky gupta

    Thanks for this post. It is really helpful.

    I just want to understand one thing –
    When you have to strategize CI/CD for an organization where they just subscribed salesforce and have one production instance only, I think here you will proceed with following steps

    1. You need to create sandboxes, full copy and dev.
    2. You need to initialize git repository ….. at this stage my question is because each sandbox contains same metadata, from where are you going to initialize git repository. What I mean is from production, from full copy sandbox or it is fine to initialized the repository from developer sandbox.
    3. On the next step we can have different feature branches from master or full copy …is it correct?

    Thanks
    Ricky

  • Where you initialize the git repo from makes no difference. Because you will be pushing from that repo to production it will be the same. It’s probably easier to do it from a freshly created sandbox but it’s not a huge difference. The next step would be to have local feature branches that get merged into master and master is pushed to your communal sandbox.