Home > On-Demand Archives > Talks >
Tips and Tricks for Avoiding and Fixing Real-Time Issues
Olaf Schmidt - Watch Now - EOC 2020 - Duration: 48:46
Today embedded systems are made up of a large number of hardware parts, SoC, CPU and networks. On the software side many layers of large software stacks, API and applications are used. The complexity of the systems is ever increasing. Most people set their focus on getting the multitude of functional requirements done. Functional requirements are what the customers sees in the first line.
But, hey, there are also temporal requirements in many use cases. Users expect a certain reaction time of their system. They don't care about complexity, well defined interfaces or big amounts of data being transferred. Press a button and immediately see a light switch on. In an autonomous vehicle the required time from recognizing an obstacle to making the decision to turning the steering wheel is only milliseconds. The requirements describe end-to-end timing in many cases. Data coming from an input has to be at output within a certain time. We call the data flow "event chain".
This talk will take you on a journey through a model-based approach. Using a model to design the system and its timing behavior has the big advantage, that it can be used in simulation. The simulation runs the model and shows the timing behavior of all components, busses, scheduling, end-to-end timing and so on. It is possible to try out different scenarios quickly and find the best configuration. In the talk we will look at both the system view and the device view. They have to be synchronized and contribute to the overall user experience. On the way timing requirements are formalized, evaluated and violation is reported. Timing requirements for individual parts of the systems like cores and software components can be derived from the model and simulation.
After determining the best configuration teams will spread out and start the implementation. Trace files, that contain the timing of the implementation are taken. The traces are tested against the timing requirements already defined in the design step. The adherence to all timing requirements can be check upon every step in the projects. Upcoming problems are found early.
Join me in the exciting journey of flashing the light of a car within the expected time.
Thanks Doini for your positive feedback.
I would like to mention that I started on a Z80 also at my first job, and I learned by decoding programs written in the assembly language for CNC machines made in Germany. Thank you!
I enjoyed your presentation, and regret to ignore this nice tool until now... it could spare some head-aches in my former projects. If the .pdf version of the slides become available, I would also be interested in getting a copy.
Hi Soma, thanks for the positive feedback.
The slides can be downloaded at the left side of this page.
Thank you Olaf for your presentation.
I'm very curious about the tracing aspect of a tool such as the one you were using. Does your tool work for example in a CPCI system with multiple boards and a PCI bus between them? Looking at the event chains how would it know that a particular packet of data came from which board if multiple boards feed different data into the same board. Is the tool RTOS agnostic and if so how does it know about the context switching and if not what RTOS's does it work with? It seems like you would have to have HW and RTOS support for tracing otherwise it would seem to me that it would be too invasive, maybe not as bad as printf's but still enough to change the timing and possibly make the problem go away (or get worse) and thus not be of any benefit.
Brett, thank you for your question.
Our tool uses the trace files as an input. It is therefore RTOS and hardware agnostic. There are multiple way to record traces and this is very hardware, RTOS, Tracer Tool dependent. You can choose what is best for your hardware, RTOS and application. We often see a combination of a small software intrumentation, hardware trace buffers and a hardware to read the buffers.
If you have multiple sources of traces (like multiple CPUs, SoC, boards...) that have been recorded at the same point in time, we can merge them. Multiple files are taken as an input and the timestamps are normalized. Of course a common element or the delta time has to be known.
We support the standard trace formats like BTF, ASCII/CSV and Tools like Lauterbach, iSYSTEM, PLS, QNX, Linux LTTng etc.
Did this anwer you question Brett?
Yes I think so - at a minimum I need to look more into the trace formats and tools. I've used a Lauterbach JTAG debugger (and Trace32 SW) but not have used their trace HW. I think one of the problems we've had on our large complex CPCI systems is we've got multiple independent clocks and thus trying to synchronize the timestamps can be difficult. Unfortunately too is that some timing problems only show up on a fully loaded system which means even attaching JTAG probes, analyzers, etc. problematic since we can't get to the boards. In those cases we've had some success saving the data to RAM and then offloading via Ethernet or whatever for boards that have Ethernet.
Thank you.
Hi Brett, I see your point. Drifting clocks and fully loaded systems are a common problem.
Some hardware (e.g. ARM STM) has high-speed debug channels, that does not add system load. So you can trace your system without -or with minimal- additional load.
Have you thought about using simulation to simulate the situation that you have? High load, drifting clocks, different CPU resources, busses/interconnects... With the simulation you may be able to understand why some problems occur and you can find an improved architecture/design/implementation.
If you like I can arrange a call with one of our technical experts.
In slide 13 you say that we should address timing issues much earlier, before the implementation phase. Where do you get the task execution times (best case, worst case) from? (At 12:17 in the video) Doesn't this imply that this can only be done on the second and thereafter iteration of a design cycle?
MartinC, good question.
Yes, the recommendation is to design the timing early in the development cycle - during the architecture / desgin phase.
In that phase you do not exacly know the task execution times. But you can estimate with the help of experts. E.g. from previous projects or from the complexity of functions.
Also we see, that waiting times (other tasks, other cores, interrupts, overloaded busses) have a much bigger impact on the end-to-end latencies and performance, that the pure task execution time.
Therefore, even if you don't have the task execution time, you can design and optimize a lot.
During the project you can learn the real execution times from the implementation and update your model and check, if every is ok using the improved model.
Another view is, that the estimated task execution times from the model become a requirement for the implementation. The designer give a maximum execution as a requirement to the implementation (in addition to functional requirements). The implementation has to fulfill the timing requirements and ist checked regularly. If the implementation is not able to fulfill the task execution requirement a negotiation with the designer/architect and other parts of the implemtation needs to take place.
So, you are right: You'll have an iteration of the design cycle.
Is it possible to share the presentation PDF? Thank you.
Hi ECU_User_001_JP: I tried to upload the presentation to this platform, but it didn't work. Send a mail to olaf.schmidt@inchron.com and I will send you the slideset PDF.
Actually, if you try to upload again, I think it should work now. Thanks.
Send us the PDF Olad and we will upload it for you, thanks.
Very nice tool. Good presentation.
Thanks for your feedback.
Thanks for the presentation. With respect to the ADC burst case study, would there be a way to save and show the actual ADC data in the TRACE? For example, perhaps the ADC went rogue because the sample was corrupted.
Yes. You can add timestamps with values to your trace file. They will be show as a separate bar (like tasks) with an y/t diagram
This looks like a very nice tool set for real-time work. Can you share approximate pricing here - e.g. hundreds, thousands, 10s of thousands USD?
I have the same question...
Hi Brian, we have a number of different products / license scope / payment models. Please send me an email to olaf.schmidt@inchron.com and I can provide a price indication.
Questions: Do you want to use Simulation or Trace analysis? How many in your team would like to use it?
Thanks for the nice presentation! I especially enjoyed the multiple case examples with different amounts of cores and tasks!
Thanks for the positive feedback. I was fun to create the examples and share it with you.
Hi;
Are the plots (33:10) generated from an Trace record of a device/OS or are they generated from simulation?
The plots from the recorded device look identical to the generated traces from the simulation.
Are there other tools, perhaps free or open-source, that can provide similar functionality to the INCHRON tools?
To our knowledge, there are tools, that cover some areas for tracing. Simulation is more complex.
Is the pricing for INCHRON approachable for a smaller startup? If not, could you point me to some those tools that would let me and my team implement some of the ideas that you presented in your talk?
This presentation seems like a sales pitch for the timing software his company produces, rather than general tips that can be applied to currently developing or existing systems.
Hi LeeT, We are sharing tipp and expertize here. The workflow and way of thinking is in the focus.
And yes: I am with a company, that sells tools, too.
If you implement the tipp and look at timing early, create good requirements, test them, analyse your traces etc... you don't necessarly need a tool. But, it may help to be more efficient.
Welcome and enjoy my presentation about the timing aspect of embedded real-time systems.
Does your system have more than 50% load? Does your data need deterministic processing? Are you thinking in milliseconds or less?
Ask your questions here or contact me at (mailto:olaf.schmidt@inchron.com) or [https://www.linkedin.com/in/olaf-schmidt/] or learn more at [http://www.inchron.com/]
PS: I started on a Z80 with 2x128 byte(!) of RAM and 512 byte of ROM. Assembler code on paper > hex code look up > hex keypad and 8-digit / 7-segment display. .... I was aged 8 or 10 and that's forty years of embedded...
Thank you very much for the presentation, it is very useful!