Home > On-Demand Archives > Q&A Sessions >
Live Q&A - Dynamic Memory Allocation & Fragmentation in C/C++
Colin Walls - Watch Now - Duration: 19:52
1) Most RTOSes - if not all - meet these primary criteria. There is no reason why they should not be MISRA compliant. If you have source code, you can verify. A number of RTOSes claim to be compliant anyway.
2) Fragmentation is not possible because there is either an available block or there are no blocks available.
3) I do not know if any, though it would make some sense. However, de-frag would introduce non-detreminism.
4) Why?
5) Doing allocation at start up results in essentially static allocation, which is a very good idea!
Regarding 4: I was just curious. You list the problems/concerns with dynamic memory around 14:51 so I was wondering if "double-freeing a pointer" could maybe also be on that list, since that seems to be a problem unique to dynamic memory.
Regarding 2: Maybe you could help me by defining "fragmentation" and how a block memory scheme prevents it?
If you called free() to deallocate some memory and then called it again, I believe the second call would do nothing.
Fragmentation is when the allocated memory is scattered around the heap, with unused gaps which are no large enough for allocation demands.
Well, fwiw, OWASP (and others) seems to indicate that the result of a double free is undefined, possibly resulting in a security vulnerability. It seems an easy enough thing to check in other implementations of free(), but maybe not something that can be assumed.
- Agree that with pools there can't be fragmentation, but if pools with right or larger block size are full, then you may find yourself in a situation where there would be enough free memory to fulfill the request, but it is allocated for smaller blocks and therefore the request fails. It is not fragmentation, but the effect for the end user is the same.
(And this just remarks the importance of tuning the size of each pool)
Yeah, this is exactly what I was thinking. But I had thought that "fragmentation" is described as the problem that results from an inability to allocate memory despite the heap having sufficient overall space, due to the fact that that space is spread out. It would seem to me that a block memory scheme is still susceptible to that problem (even if it's intentional) and therefore doesn't solve the problem of "fragmentation". If I partition my heap into 1024 blocks of 16 bytes and then make 1024 requests for 1 byte each then I've exhausted my heap despite only using 6% of the available memory; shouldn't that qualify as "fragmentation"? Perhaps a block memory scheme helps limit the amount of fragmentation that could occur, but I'm struggling to see how it eliminates it.
The final effect is indeed the same - despite counting a grand total of enough free memory, the system can't fulfill the allocation request for how the free memory is organized.
Fragmentation refers to a condition in which free memory is fragmented and interspersed with in-use memory. This condition is not possible when using pools as described by Colin.
Though there is no magic wand to solve dynamic allocation :-) . Pools need to be carefully designed (and tuned) both to limit the case you describe (slack space at the end of a block) and the case I described (wrong dimension of pools with respect to the use).
- In Windows 3.0 / 3.1 (and possibly earlier versions) the allocation function (GlobalAlloc/LocalAlloc) returns a handle. In order to access the allocated memory you need to Lock the handle and Unlock it when done. This allows the system to move memory around when it is not locked by any process. BTW if you needed memory to stay fixed a dedicated flag was available in the allocation function.
Is there a library, tool or example technique that will help developers visualize the heap fragmentation in a dynamic execution environment? For example, is there a heap profiler that can run alongside the application that will show the state over the heap every 10 ms or every period that's configurable?
I am not familiar with such a tool. I guess it would need to be run as a separate task or timer tick interrupt service routine.
Oh, also I had read on Embedded Artistry about stack protections offered by GCC and Clang and thought I'd post a link for others.
11:36:10 From Levy Gabriel : What's the impact of fragmentation in sensor data stored via DMA? What are the strategies to overcome this issue in C/C++? 11:36:25 From Nathan O. : If you don't know the size of the data you have to process and use dynamic allocation, are you not just pushing the problem away and hoping for the best that it won't reach over the capabilities of the system ? 11:37:04 From Naishadh K : Great talk, Collin. Suppose, we are careful of freeing the memory after malloc. Are there any other ways memory leak can happen? If yes, what are the things that needs to be taken care of? 11:37:31 From Levy Gabriel : Let me rephrase: fragmentated memory 11:37:39 From Levy Gabriel : The DMA data is continuous 11:37:58 From Jim Norton : I didn't watch the talk. But I do wonder if there has been any work/research towards creating a system that can defrag memory in a low priority task that can still keep near hard real-time requirements? 11:38:40 From BobF : Re: Choice between Static/Dynamic. Presumably, the life-time of the product also potentially plays a part .... how well it's originally designed and whether subsequent field support i.e. OTA is serious! 11:39:33 From doolittl : When dynamic memory is used, do you have a favorite tool or similar strategy for debugging fragmentation (or leak) issues? 11:42:07 From Gonzalo : Which cases really need to use dynamic allocation? In power electronics and implementation of current control systems I have always avoided dynamic allocation because the real time capacity is absolutely needed, otherwise something will blow up. Anyway, now I wonder how to use dynamic allocation securely if needed to avoid depriving me forever not to use it in other applications. 11:42:50 From Nathan O. : Any recommendation on how to define the heap size ? Should we assume max load / size for everything ? 11:43:13 From John : Can you elaborate a bit more on how overflowing, and especially *underflowing* the stack would happen? Because isn't the compiler at fault for these (not the programmer)? Which doesn't actually solve the problem of course but does impact how these issues can be fixed. Thanks! 11:45:20 From John : If you are on bare metal, and don't have an RTOS, any recommendations? Should static alloc be preferred more strongly? 11:45:20 From Nathan O. : Répondre à "Any recommendation..." Then you have to carefully define the size of your stacks ;) 11:45:52 From Jim Norton : Suppose we could create an allocator that returned a handle to allocated memory instead of a direct pointer. Would that get us closer to being able to have a defragment system in languages that use pointers? 11:47:20 From Jim Norton : Overflow is easy to create with recursion 11:47:52 From John : Cool, thanks! 11:49:18 From John : Basically, write your own garbage collection, right? 11:49:32 From Jim Norton : :-) 11:50:03 From Keith J : Thank you Colin 11:50:14 From Jim Norton : Thanks guys! Appreciate it 11:50:39 From Charles Miller : Thank you, gentlemen. 11:50:52 From Naishadh K : Reacted to "Thank you Colin" with 👍
Thanks for the great talk, Colin! I have a few thoughts/questions for you about it: