by Bindu Wavell, Chief Architect at Zia Consulting
There are two reasons I decided to write this post. First, I want to acknowledge Alfresco for their recent investments in the developer ecosystem. The other reason is to explain where I think we are heading with our development efforts. My ulterior motive is to find people to collaborate with us on these efforts.
Since Thomas DeMeo joined Alfresco as VP of Product Management a bit over a year ago, we’ve noticed a dramatic increase in the focus on system integrators being key stakeholders for Alfresco—and not just based on expertise in sales and business development. After the release of Alfresco One 5.0 at the Alfresco Summit, we saw the likes of Peter Monks and Gabriele Columbro tapped to bring focus to user stories that are important for administrators and developers within the product management organization. Recently, Richard Esplin transitioned from the community lead to focusing on the Community Edition within product management. Alfresco hired Martin Bergljung and Ole Hejlskov to focus on developer tooling/evangelism and community outreach. Within weeks of starting, these individuals put together a new release of the SDK; incorporating contributions, adding new capabilities, and completely revamping the documentation. I’m thrilled that Alfresco is focusing resources in these areas because I think we will see resolution of a lot of technical debt—and that allows for better solutions in less time, leading to a bigger and more vibrant community.
In the past year or so, Alfresco engineering has begun to reorganize into smaller, more agile, scrum teams. This reorganization—along with the focus on product management—will drive initiatives like release agility to provide more frequent and better tested releases of distinct products. It should also provide a platform for resolving technical debts in a more sustained and predictable fashion than we’ve seen in the past. We can also expect cool new products that are easier to integrate and customize. Things like Activiti Enterprise—the integration between Activiti Enterprise and Share—enhanced Office services, reporting/analytics, media management, and even new Case Management features. Not to mention, significant improvements in the repository, Share, and records management.
As the Chief Architect at Zia, part of my mission is to facilitate improvements in developer productivity and satisfaction. In addition, I want to help the team find ways to improve project quality and consistency. I’d like to share where we are heading in these areas, but first let’s cover where we’ve been.
In the past year or so, most of our projects have been based on the third major revision of our development framework. We call the framework—the project structure and the associated tooling—Zia Alfresco Quickstart (for more information, watch this video). Quickstart includes a standard project structure that we evolved from the all-in-one archetype provided by the Alfresco 1.0 Maven SDK. It features reusable code, examples, best practices, and, to some extent, standardizes how we version and deliver our projects and reusable sub-projects.
With version 1.0 of the SDK, as well as our earlier project structures, we were seeing cycle times (from the point when we saved our code to when we were able to exercise the code) of between two and five minutes on very powerful laptops with lots of RAM and solid-state disks. One of the main reasons we started evolving the SDK was to reduce this cycle time. When we started using Quickstart for customer projects, we were able to reduce the cycle time for most edits to about 10 seconds. We did this by taking advantage of incremental compilation and hot deployment techniques. If I was writing this post a couple of years ago, it would have been all about flow. It was hard to experience flow when you had time for tea and a bagel after most code/config changes. Fortunately, this is not as much of an issue anymore. The Alfresco 1.1.x SDK made some similar techniques available for the wider developer community. With the 2.0 SDK, this has been improved even more—but there’s still work to be done.
One area where Quickstart enhances the SDK capabilities is an integration testing framework for repository customizations that also supports continuous integration and, to some extent, delivery. After we presented this framework during Tech Talk Live #69 (see video above), the 1.1 SDK added a similar capability—however, that solution has been a bit unreliable. We contributed the Quickstart testing framework to the SDK team and are hopeful it will be incorporated in the near future. We are excited that the 2.1.0 version includes support for the Share Page Object testing extensions to Selenium WebDriver that was, and continues to be, developed by the Share engineering. This will make it much easier to create tests for UI customizations and to make sure our customizations don’t unexpectedly break existing capabilities provided with the products.
With the project structure we used before Quickstart, it often took us between four and eight hours to get a full development environment (just the Alfresco pieces) installed and configured. With Quickstart, we’ve reduced this to around two or three hours.
We often need to work on code for multiple projects in any given week. In order to handle this, and to accommodate customer variations, we usually set up our development environments in Virtual Machines (VM). Nearly every time we’ve had to start from a base OS machine.
Typically, one team member sets up the initial VM, installs the development tools, and sets up the project structure. Then the VM is shared with all of the team members. We make heavy use of VM snapshots and usually someone keeps a pristine copy of the VM that tracks releases. Should a new developer join the project, or an upgrade be performed, we utilize this pristine copy. Often these VMs are over 40GB, requiring a substantial amount of time just to copy the data.
At Zia, we’ve been testing a few different code review approaches. Some projects are doing regular reviews (weekly for example), others are focusing on reviewing each new significant feature. The ability to create pull requests from forks and branches in BitBucket and GitHub has provided enough of a framework for us so far—though we’d love to incorporate more tooling around code quality and coverage to provide consistent feedback to users.
The Path Ahead
Quickstart has been seen as a proprietary solution that allows us to complete projects faster and at lower cost than we were able to previously. One of the downsides to it being proprietary is that there is a smaller community for collaboration and support of the approach. The next version of our project structure is being developed in-the-open using the open source model we admire so much.
The Quickstart project structure is quite different than any of the official SDKs, and there are good reasons for the differences. In many cases, they improve on what is available in the community today. However, what we have is different enough from the standard that new team members often have a steep learning curve to become proficient and ultimately master the structure. So, while a seasoned practitioner will be very productive, newer folks require more time and support to become productive. This turns out to be detrimental to the goal of improving productivity for some team members.
With the next generation project structure, we plan to stay closer to the official SDK so that there is a much larger community for collaboration. While we still plan to include support for certain opinionated features, we will also support and default to using more traditional Alfresco implementation approaches. Our hope is that this change in direction will facilitate quicker onboarding and allow SDK and Alfresco upgrades to be handled more expeditiously.
With Quickstart and all of the Alfresco SDKs to date, we have to duplicate a large portion of the boilerplate code for common Alfresco customizations such as web scripts, actions, behaviors, jobs, and workflows via cut and paste. While most of these aren’t difficult, they do tend to be error prone.
Our new approach is to provide a Yeoman project generator. This automates the construction of a project using the all-in-one archetype from the SDK while adding a few bells and whistles for improved productivity. Though it’s still in its infancy, this part of the project is available now and we are using it for customer projects when appropriate. In addition, we are working on sub-generators for common and boilerplate things like: adding amps (source, third party from git, third party from Maven, and checked into the project), adding webscripts, and adding actions. We also plan to work on sub-generators for adding behaviors, models, workflows, jobs, javascript extensions, and data bootstrapping. We may even create generators for common tasks like switching from h2 to a real database, enabling ldap, hooking up the standard image processing tools, and other common tasks that collaborators think will add significant value during the development phase of Alfresco implementation projects.
The development VMs we’ve been using are difficult to version control, slow to copy, and frankly, take significant CPU, disk, and memory resources that we’d prefer to allocate to development and runtime tasks.
We’ve been toying with setting up our development environments using devops tools such as Ansible, Chef, VMWare, Vagrant, and Docker. Using Docker, we have been able to spin up and exercise clustered Alfresco environments on a single machine for testing and POC activities. We’ve also used Vagrant and Ansible to get about a 40% head start on our development VMs. The hope is to script 90% of the project setup efforts, to reduce project setup time, and increase consistency between our projects. We also hope to utilize Docker or other lightweight container solutions to reduce the overhead of our environments.
To date, we’ve had mixed success using these tools to setup our development environments. It often takes a significant amount of time to create and refine the devops scripts and we don’t expect to see a return on our investment until we’ve utilized and stabilized these tools with a number of projects. Fortunately, we have worked with a few customers to create production quality release/delivery substrates using these tools. Our hope is to incorporate our experiences from these projects into the developer tooling with an eye toward standardizing how we install and configure Alfresco solutions in all environments. We feel that by utilizing these techniques, developers will be able to rebuild small, containerized environments from scratch when needed, rather than maintaining and sharing monolithic VMs. This approach will be much easier to version control, easier to upgrade, easier to share, and will be lighter on resources.
An area we are also exploring is the use of cloud development infrastructure (e.g. Codenvy) to develop, run, and test our projects. We’d like to utilize our devops work and create containers that we can use during development and testing and potentially as a vehicle for delivering projects as well. It would be great if this allowed additional interactivity and collaboration during code reviews, while fixing bugs, and for training/coaching users one-on-one or in groups. We’d also like to reduce expenditures on hardware for developers and to deliver progressive capacity to our engineering organization. The ultimate goal is to work smarter with our in-house, remote, and offshore team members.
Our first usable effort in the area of cloud development is the contribution workflow for our new Yeoman generator. By clicking a button on the project GitHub page, we can provision a development environment that has access to the project source code and a docker container that has been set up with the appropriate versions of Java, Maven, Node, and Yeoman. It would also have the local (in the container) Maven repository pre-seeded with assets needed for compiling and running the Alfresco projects we build while testing out the generators. Someone wishing to make a contribution can start developing and testing in under a minute and can send us a pull-request directly from the generated project on Codenvy.
View a generator contribution demo video here:
We’d like to invite you to collaborate on these ideas and deliverables. Currently, we are focused on completing our first pass on the Yeoman generator and some high value sub-generators. We’d love to collaborate in order to continue evolving the developer/implementer experience for Alfresco extensions. If you are interested, please leave a comment, send an email, or ping me here on IRC. Once the generator is in good shape, we’ll likely set up a cloud-based development experience. This will be driven by the generators and backed by pre-packaged containers that can be used in the cloud, on our development machines, and possibly in customer dev, stage, and prod environments. Imagine quickly packaging your configuration and customizations with Alfresco into an all-dependencies included container. You could then run tests against the container, deploy that tested container to stage, perform UAT and—assuming everything is accepted—promote the exact same (tested and accepted) container to production.
Now that’s the future of Alfresco development.