Key Takeaways

• Large Language Models (LLMs) provide a great opportunity to accelerate requirement development
• The “Docs as Code” paradigm and tools like StrictDoc support a low-cost and lightweight approach to requirements traceability
• Open-source projects are awesome to sharpen technical skills if you don’t have the opportunity to practice in your day job

My Exploration into Open Source Functional Safety

I recently created my first Pull Request for the Zephyr Project. I’ve been eager to contribute to an open source project for quite a while now. Finding the time has always been a challenge and most projects I looked into had a steep learning curve to effectively contribute.

Things finally fell into place for me to explore and I found the Zephyr Safety Work Group. They are working to achieve ISO 61508 SIL 3 certification, which will allow companies to quickly use Zephyr in Functional Safety applications across many industries. This was perfect since I’d just completed a project where I led a team to achieve ISO 61508 SIL 2 certification of their radar sensor. I found my new project!

If you’re not familiar with Zephyr, it’s an open source Real-Time Operating System (RTOS) used across in thousands of embedded software devices around the world. They provide the software layer that helps to tie the hardware functionality to a product’s application. It’s responsible for making sure everything happens at the right time, in a predictable and controlled fashion.

So How Could I Contribute?

After reviewing the Zephyr Requirements GitHub page, I found a list of tasks (Github Issues) related to their Functional Safety development activities. It seemed the easiest way to start contributing was by writing requirements for the kernel software features.

One of the things that really called out to me about this project is that the team was using Github and an open source tool called StrictDoc to manage design traceability. This was a similar approach I’d followed on my last project where we needed a low cost tooling approach. Using this combination of tools has a lot of benefits:

• Docs as Code: Writing documents in markdown allows the requirements to use common code development practices like change requests, code reviews, and version control in a way familiar to software developers
• Low Cost Tools: Popular industry standard tools cost tens of thousands dollars per year, open source is free (though I’m sure contribution is appreciated)
• Tool Verification: StrictDoc is developed with full design traceability and testing which supports easier software tool qualification

This was an excellent opportunity for me to test out some ideas I’ve been thinking about regarding using Large Language Models (LLMs) to assist with writing tasks. It also gave me the chance to tick off several items from my “Things to Explore” project list. A few people have asked about my experience, so I thought this post would be a great way to share and use the opportunity to practice writing.

Writing Requirements Using an LLM

Generating a Baseline Set of Requirements

So how did I start? A friend had recently shared a free Perplexity Pro membership with me so I decided to their Deep Research functionality a try. I’ve dabbled in prompt engineering, but I’m no expert yet.

I tried the following prompt:

Create a list of software requirements for the Zephyr Mailbox module of the Kernel Services firmware. 

Use INCOSE best practices and follow the EARS format. The Zephyr Safety Requirements should be used as guidance when writing these requirements. 

You should also consider the test scripts written for the Mailbox module as those should be testing the requirements that you'd be writing. Please ensure as high of requirement to test coverage as possible. 

You can use the requirements written for Semaphore and Interrupts in the Zephyr Safety Requirements repository for examples of what the requirements should look like.

This gave me a great starting point. I couldn’t believe the LLM automatically found all of the relevant website content by itself. It gave me a list of requirements following the EARS format and neatly grouped them into functional sections.

It made sense that the requirements generally followed the organization of the Mailbox Documentation. What surprised me was that the categorization actually worked pretty well. There were a few times where I saw requirements in a functional group created from content in later sections of the documentation.

Moving the Requirements into StrictDoc

Next, I needed to convert our project requirements into a specific format used by StrictDoc. I thought, “Why not see if the LLM can lend a hand here too?” To my Next, I needed to convert the list of requirements into the specific format used by StrictDoc. I thought, “Why not see if the LLM can lend a hand here too?” To my surprise, it was able to get pretty close with a simple follow-up request or two. Unfortunately, I don’t have the exact responses I used anymore, but it was something like this:

Update the requirements you've provided to follow the StrictDoc format used in the other Zephyr requirement documents

The LLM did a pretty good job of putting the list of requirements into the sdoc format, there were just a few things I had to tweak to get it to build:

1. Requirement IDs: These weren’t following the format I’d initially planned
2. License Attribution: This was missing from the generated file
3. Requirement Fields: Not all necessary fields were included and filled out consistently

After making those changes I had my initial commit. You can check it out here: link. I could have asked the LLM to clean up more, but at this point, it seemed easier to just finish up manually.

Polishing the Requirements

Now that I had a baseline, I had to make manual updates to get the requirements to a ready state. Considerations for my manual review:

1. Consistent Phrasing: Update requirements to use standard terms and phrasing used in other requirement docs
2. Correcting the Subject: To improve readability it’s important to ensure the subject is consistent across requirements
3. Sanity Checking: Ensuring the requirements made sense, aligned with the documentation and removing hallucinated (fake) requirements
4. Missing Requirements: Create new requirements while reading the documentation if described functionality didn’t relate to an existing requirement

Finally after making all the changes and confirming I had everything updated to meet the Zephyr Contribution Guidelines I submitted my Pull Request.

How Was Using an LLM To Generate Requirements?

It was amazing how close this tool got to giving me exactly what I wanted.

On a personal level, it was really reassuring that I could see right away what was missing. I’m regularly concerned that over-reliance on LLMs will cause me to miss too many details.

From a technical standpoint, StrictDoc’s build errors were able to help me quickly sort through the syntax issues. Even without prior experience with the code, I had a great starting point after only 20 minutes of prompting and review.

This activity worked well for Perplexity due to two main reasons:

1. Easy Access: The Zephyr documentation is publicly accessible online
2. Quality & Completeness of Info: The documentation was well-written and quite thorough, covering most aspects of how each module works

The LLM could understand and use this information effectively because it was available online and provided clear, detailed explanations about what it needed to know.

I also attempted using a locally hosted LLM, but the outcomes weren’t as impressive. That’s a substantial project idea on its own, which I hope to find time to explore further. There are numerous techniques to enhance local requirement generation that could enable its use in a private setting.

So, What Were the Results?

There’s still some ongoing discussion about what changes are still needed to complete the Pull Request. I’d love some additional review feedback if anyone is interested. Most of the feedback so far has been around gaps in my understanding the system I’m defining and the practices followed by the safety team. A couple requirements got deleted. They made sense in isolation, but not on a fresh read with more context considered.

All in all, I found that using an LLM to support requirement creation quite successful. I spent a lot more time getting up to speed with the Open Source Project – understanding their mission and objectives, familiarizing myself with their pull request guidelines, learning how to participate in meetings, and so on.

Writing 24 software requirements took me around 3-4 hours. I think that’s pretty solid considering I didn’t have past experience with the code. It’s challenging to predict how well this approach scales, but I’m eager to give it another try soon on some of the other software modules.

General Lessons Learned & Insights

LLMs can give a strong jump start to write requirements if you feed it good design documentation

• High quality documentation is a critical factor
• Local LLMs would probably work, but it seems like a lot more refinement techniques are needed to get something comparable to online tools

The “Docs as Code” paradigm works well as a low cost way to version control requirements

• Several open-source tools support requirements management & traceability
  • StrictDoc seems user-friendly and simple to set up
  • OpenFastTrace worked well on a past Functional Safety project
  • Sphinx-Needs is another tool that looks interesting but I haven’t had the chance to dig too deep yet
• Github did it’s job as a git support tool, but it handled the Pull Request Review feedback and markdown changes somewhat poorly

Working on open source projects is a great way to practice skills you don’t have the opportunity to do in your day job

• There’s something for everyone out there!
• I hadn’t thought to find this type of project, I always think of straight code development projects when I hear Open Source Software

---

Need help with requirements, Functional Safety, embedded software, engineering processes? Let’s discuss how I can assist you.