How Agile is Changing the Face of Embedded Software Development
This presentation is ideal for anyone who is either new to Agile, considering using Agile or even has experience in working with Agile methodologies and practices with embedded software or firmware developments.
It will clarify the Agile landscape, covering both process based aspects, such as Scrum and various techniques, including Test Driven Development (TDD) and some of the underlying foundation principles, such as Continuous Integration (CI).
As part of the discussion, we shall look at some of the modern-day tools that help apply Agile techniques(e.g. Docker) and finally look ahead to the current gaps and where embedded systems offer particular challenges to the use of Agile techniques.
According to Niall Cooling, what is the primary reason Docker (containers) became important for embedded continuous integration?
Thanks, Steve, yes maybe I should be clearer about that.
No worries. My IT manager was just saving money and got me Windows 10 home on my laptop. These things happen.
Great presentation Niall!
Some considerations which are particular to embedded systems development which makes it that much harder to apply Agile and make it pay off are:
Dependencies with hardware releases, fixed delivery dates, inadequate software tools, limited adaptation possibility due to hardware costs... and culture!
What are your experiences and thoughts about these and how have companies overcome?
Hi Lance, (full disclosure Lance and I did our degree together a long, long time ago ;-) Many companies I've worked with are still trying to work out the RoI of trying some of these techniques. The underlying problem is that we typically have no quantitive measure of process improvement when it comes to software (it's all based on "gut-feel").
Culture, as you mentioned, is the biggest barrier to change. Most "embedded" companies come for a manufacturing background that measures productivity and process very differently to software-centric systems. I believe there is still plenty of research required in this area.
Niall.
Thanks, Niall I agree - also the line between hardware and software is becoming more blurred with the advent of FPGA's and software configured hardware so perhaps Agile will find more relevance for the next generation of embedded engineers. Ultimately all product companies are chasing greater and greater efficiencies and productivity.
Yep, it was a long time ago that we did our degree together and I am very grateful to have met exceptional engineers along the way as well completed a degree course which was ahead of its time.
I am in embedded development too. As I agree with most of your arguments, I am puzzled as to why "fixed delivery dates" should be a hindrance for Agile development, Agile done as it was intended will actually help you to keep those deadlines, by using the agility to handle the unforeseen obstacles that will pop up.
I would say that the biggest problem in getting the agile mindset in embedded is the wish to get frequent feedback from the customer on each iteration is hard, as much of the embedded code is not visible for the customer, the trick however is to redefine the "customer" seen from SW development the customer might also be some internal dept. example I am developing the internal control of the hardware, most of my changes in mainly seen by the developers making the interface to the world, that be the display or a field bus. So in many cases they should be the customer.
But this redefining of customer, and creating the culture of feedback in an organisation is often a bigger challenge than one might think.
I am rambling now sorry, i will finish.
Hi Niall,
Great presentation and I liked how you balanced the pure Agile with constraints of embedded systems. When I was working in Aerospace we had different teams try different approaches with varying degrees of success. Although the tools available now are much better/easier (I think) than they were just a "few" years ago. ;-) I will check out your website and youtube channels! Thanks!
Hi Andrew,
Certainly, the tools are improving, but I still see a "chasm" between good-old unit testing and integration/system test when dealing with embedded systems. Some companies are starting to wake up to this and realising there is a business opportunity here. It will be interesting to see what offerings we get on the back of people having to work remotely. Thanks for the feedback. Best regards, Niall.
Great talk Mr. Cooling! You provided valuable information in your talk about agile within embedded world that is really enlightening. Thanks!
Thank you for taking the time to watch. I'm glad you found it useful.
Hi Niall,
Thanks for a very insightful review of various tools that you use yourself at Feabhas and no doubt recommend to your clients. The likes of Docker, Git, Blue Ocean etc. are clearly helping teams build, test and integrate their systems with greater efficiency and in less time. Obviously there are challenges in the embedded world when trying to operate with fully cross-functional teams and extend CI/CD to the target test environment; plus the issue you raise of node-locked licensing vs. containers is arguably one of the top tips from this session.
I must say though that I was a little uncomfortable at times with the Agile theme running through the presentation, in particular imbuing certain tools and techniques with Agile characteristics. Firstly, the Agile Manifesto values “individuals and interactions over processes and tools”; thus, the use of any tool is already a compromise. Secondly, isn’t a tool just a tool? Doesn’t Agility come from judicious selection and use of a tool such that there are improvements in efficiency, quality, collaboration, and so on?
One example is slide 26 where you say: “Git is definitely the dominant VC system, certainly from an Agile perspective.” I agree Git is definitely dominant right now but I think that’s mainly due to factors such as the widespread use of Github, Atlassian’s seamless integration between Jira/BitBucket/Confluence, and plenty of other 3rd party integrations for Git. Subversion may have fallen out of favour but I don't think that has anything to do with it being somehow “non-Agile.” Branching and merging is generally the most fraught aspect of any VC tool and that includes Git. Keeping these cycles short, very much aided by stories/defects with estimates of hours rather than days, is the best way to avoid conflicts and keep the CI engine running; i.e. it’s all about process, not tooling.
More generally, a question you ask several times is “how Agile are you?” So do you feel that Agility has become a tickbox-style progression model whereby you can be Level 1 if you are using a tool which captures User Stories, Level 2 if your team has adopted Scrum, and so on? And if your CI engine is generating a fully integrated and tested build every night, your team has achieved full Agility and can rest on their laurels? My worry is that this sets us on a course for AINO, a situation you highlight very nicely on slide 8. A team may claim Agility through using Jira however what if the stories are poorly written, what if estimates are inaccurate or missing, what if the backlog is seldom groomed, what if the server goes down and takes several hours to restart? A team may claim Agility through using Git however what if a feature branch gets mired in unit test failures and doesn’t deliver by sprint end, what if some scenarios of a new feature do not have unit tests (a common problem with User Stories that Use Cases tend to cover), what if a Pull Request fails due to a merge conflict?
It’s now nearly 20 years since the Agile Manifesto was published. Furthermore, the term “Agile” has been used in all manner of contexts… and perhaps over-used, to the point where arguably it is falling out of favour. Thus when you’re talking with clients, running training courses or engaged on consultancy work, does “Agile” still resonate? Is the word greeted with enthusiasm or resignation? Personally, I seek to avoid the ‘A’ word and instead focus on the values and principles which, to be honest, have been around in one form or another for much longer than 20 years. This frees up any recommendation of a tool or process to focus on how much it can improve delivery times, quality, and customer satisfaction rather than whether arbitrarily it is or isn’t Agile.
Once again, thanks for a very interesting session.
Hi Brian,
In only an hour it's hard to work out what not to talk about. You've summed up my intention of the talk very well, so thank you for that.
One the first point, tools are essential as they balance the triangle of tools<->people<->processes. Unfortunately, late '90s companies were very guilty (as were managers) believing that buying the latest 'tool' would suddenly solve all their problems (think UML). That has now flipped to thinking 'going Agile' will solve all their problems; and yes, we should be wary of that move.
Many of the tools that have come out of the desire to be more agile have come out of necessity. A good example is TDD frameworks. They are great at the unit test level, but in the embedded space there is still a need for comprehensive system-level test tools (though there seems to be a gap when we look at integration testing). More importantly, tools don't sit in isolation. A good example is Git and the pretty seamless integration with modern build servers.
Branching and merging is always a nightmare, but, certainly from my experiences, it was the change to the branching philosophy with Git that helped. It took me some time to reset my mental model when I moved from Subversion to Git (similar to moving from C to C++).
I always talk about 'agile' and 'Agile'; Agile is becoming a 'tick-box' exercises by management, whereas 'agile' is fundamentally about addressing process improvement and a focus on quality. Some of it is 'well we've always done that...' but there are some key areas that are 'that makes life easier...'. Ultimatly it should be viewed as a loose framework for each company/project/team to adapt - it's not a drop-in solution (as people seem to think Scrum is).
Thank you for your comments, they are very much appreciated.
Hi Niall, Good talk on Agile. On TDD... it starts as flossing, i.e. something you should do. I've flossed most my adult life. It's a habit. I rarely don't. TDD is not like that. With TDD skill develops and soon TDD becomes FUN.
Hi James, I can only talk about my experiences when consulting with companies who have/are trying TDD. Hopefully, you saw I pointed people at your presentation on TDD and put the rest of us to shame about presentation style ;-) I'll up my game for the next one!
Hi Niall, I really enjoyed your presentation. I appreciated the recognition of working in multi-disciplinary teams (SW, HW, Mech) when it comes to embedded systems development. So many books focus on just agile software development. Also good coverage of Docker, as I’d always assumed this is a Web/Linux thing that wouldn’t apply to the embedded world.
Thanks. Docker is now central to so much of what we do internally. It's shown its real benefit as we've moved from gcc-5.4 all the way to gcc-10 (mainly supporting C++11/14/17/20). It's allowed us to test our codebases in a painless way without impacting the existing setups. Hopefully, the blog postings about Docker will give help cement its applicability.
Very good presentation Niall. Thank you
Thanks for taking the time to watch
How do you work around some of the challenges you described, e.g. having to deal with hardware/software integration when hardware is part of a different team that does not participate in the software team's sprints? How do you align with hard dates on the hardware integration schedule?
And there in lies the biggest challenge. There are no easy solutions out there. A couple of companies have reorganised the whole team structure along project lines, so all members (h/w, s/w, mech, etc) are all part fo the same team rather than having a separate hardware department and team. But that is only feasible will smaller companies and dynamic management. Other (larger) companies are using the SAFe framework to try and address this issue. This is where better simulation/emulation technology is needed to mitigate schedule risks.
What are some examples of integration tests that the CI would be performing ? Is the CI doing these within a software only domain (host server) or is this being run on the target (via jlink/debug interface and additional hardware to monitor pins/voltage levels/signals etc.. ) ?
The majority are S/W only (as that's the easy bit). More complex/complete CI setups are doing full h/w-in-the-loop testing; but that typically requires building custom rigs (the best I saw used Lego!). I've seen Raspberry Pi's used as well to interact with voltage signals, etc. It's something we're working on at the moment to try and simplify.
When you talk about integration tests that happen in the S/W domain - what type of tests are being conducted ? Can you provide a hypothetical example ? Is it just as simple as the interaction between between modules instead of the just limiting to just a module (unit testing) ?
To the greater extent yes. Integration testing should be focussing on the Module/Component/Subsystem API protocol. What's typically missed is that almost all interfaces have some sort of protocol (based captured as a state machine). Good integration testing is about state-space investigation of this protocol. Fundamentally Mocks are there to prepare a unit for integration testing (as opposed to stubs for unit testing).
Hello... Thanks for the session.. covers a lot of stuff... had two questions related to scrum team... what if the scrum team is unbalanced in terms of competencies and experience... secondly how to best estimate exploratory or R&D type of stories, like working with some new hardware etc
Two good questions. On the first one, an unbalanced team, I have seen (in real life, on a real commercial project) pair programming help address this. Done well/correctly, pair-programming has so many benefits. Unfortunately, it's really difficult to get management buy-in. The second one is far more difficult; especially where there is more R than D. This is why I prefer lean over scrum, as it's a pull model rather than a push model (I see scrum as a push model). It is all related to risk management and trying to decompose the core requirements. Check out https://www.youtube.com/watch?v=pz8CpSUiULo for some ideas
Good stuff. It was handy to see you break down working across HW, SW and Mech boundaries with a CI model. Don't see that topic come up very often. Thanks for the overview of a CI build-test-deploy strategy and some tools that could be used to make it happen.
Thanks for the talk. I'm going to discuss this talk at my client's company.
Thanks. We do have a full day covering all things agile in more detail which I'm happy to discuss offline. Glad you found it useful.
will the slides be uploaded?
Yes, they should have been. I'll check with Jacob why they're not there. But feel free to ping me directly (maybe on LinkedIn) and I'll send you the PDFs if they're not up by the end of the day
Got 'em. Thanks Niall for the wonderful talk! Lots of good nuggets in there. The title tricked me a bit, but you are right, I once said to a team at my company "if you're not doing CI (and we're not yet in many cases), you're just moving stickers". I've been on teams before that have done CI and really saw the benefits in the "agile mindset". I along with a few colleagues of mine are pushing for more to go down the CI path, and I liked all the slides you had on the topic. I'll be referring back frequently.
Very nice presentation. Thank you
Thanks for the feedback
Thanks for the talk! I am going to try to bring continuous integration and TDD into my student team, and this was very helpful
Thanks Markus. I've posted some work on CI, Docker and embedded on our blog at https://blog.feabhas.com/2017/09/introduction-docker-embedded-developers-part-1-getting-started/ might be worth a read.
Great presentation Niall. You mentioned near the start that you're not a great fan of stories. I'm wondering what method you prefer to capture the business and technical aspects? Are you referring to traditional requirement statements such as "the light SHALL turn on after 200ms", etc.
Hi Alan,
Thanks for the feedback. I think it very much depends on your application, but anything that had any degree of integrity must have some sort of traceability back to the "SHALL" (dull I know but necessary). Check out a webinar I did last year which goes into a lot more details and cover user stories, use cases and other techniques: https://www.youtube.com/watch?v=pz8CpSUiULo
Thank you for the feedback and link, Niall. I always had a niggle about agile in this respect.
Enjoyed this talk. Thanks Niall.
Quick word of warning about Docker (from 45 minutes into the presentation) - it only runs on some versions of Windows 10, so make sure you have Win10 Pro. (Yes - I got caught out by this a few months ago!)