Search Posts

Linux memory management (insufficient memory, memory leak detection / prevention)


Linux memory management (insufficient memory, memory leak detection / prevention)

I think that it is often the case that the memory of PC used for server etc. is enough.If you check it out, the lack of free memory and the appearance of memory overflow in SWAP etc.Anxious things come out in various ways.

Various factors are intertwining the cause of slowing down the operation of PC,Sometimes the symptoms that appear on the surface do not indicate effective measures.In the following, we will concentrate on memory-related explanations.

  1. Check memory status
  2. Should I have enough memory-can I afford it?
  3. Which process is consuming memory?
  4. How to detect memory leaks? How to prevent?
  5. Should I use SWAP if memory is enough?
  6. Well, how do we know the real memory shortage?
  7. How much memory and SWAP space should be reserved

Check memory status

The way to check memory usage isThere are free, top, ps, vmstat, / proc / meminfo, etc.You can use xosview or ProcMeter to view the situation graphically on X.If you check the memory usage status with these,Looks like chronic memory-lack,It may be puzzled.However, judgment as to whether memory is insufficient, what is the cause, etc. can not simply be made.

Should I have enough memory-can I afford it?

Oh? Is the remaining memory-only 1.6M (free is 1628)?The display unit of free is 1k blocks,It looks like there is only a margin of 1.6M. Is it a pinch?

Look at-/ + buffers / chche

Under Linux (or UNIX?),I have allocated the memory to each processUsed for buffer and cache,It reduces the load on disk I / O.Therefore, the remaining memory (free) seen by the free command etc. is about 1MIn many cases, there is only a level of dealing with the momentary use.

In this example, the substantial remaining memory isConsidering the amount diverted to buffer and cacheIt can be calculated by free + buffers + cached.In the above example, free + buffers + cached = 1628 + 20112 + 126848 = 148588.There is still room for memory.

Which process is consuming memory?

Whether individual programs are wasting memory, what to do if you want a rough look.

The free command only gives the overall memory usage.Use top or ps to check the memory usage of each process.

  • If you start the top command and then “M” (upper case), it will be displayed in the order of memory consumption.

  • You may use ps command as ps -xl –sort -vsize.

How to detect memory leaks? How to prevent?

Memory-related problem classification

  • Memory leak: Do not free space. Forget free.It causes the operation error due to lack of memory.

  • Wild pointer: A pointer that refers to trash.Cause of memory data destruction (memory smash).

    • Uninitialized pointer.

    • Error in initialization, error in pointer operation,Overruns of pointers, contamination from other wild pointers, etc.

    • Dangling reference: I will access the open area.The timing of free is too early.The timing of using alloca is bad.(Eg fgets (p = alloca (10), 10, stdin) and so on …)

Memory leak detection

There are the following methods to detect memory leaks.

The importance of malloc and free verification is thatWith a program (such as server software) that continues to operate for a long timeIt means that a small amount of memory leaks will build up and cause malfunction.Slow memory leaks are difficult to identify without tools for detection.

For programs that end in a short amount of time,Even if you leave the memory leak, it will be collected by the system at the end, so there is no need to worry about it.It is a problem to say that it is a problem submission of a school or software that is delivered to a customer.Think of the OS as memory management software.(Note. According to the ANSI / ISO C standard, it seems to be OS dependent whether or not the area is properly released in such a case.)However, it will be difficult to reuse the loosely assembled software based on this idea.If you have a “rapid” memory leak that causes problems with such programs,Even if you do not use a special tool, you can check it just by observing it with the top command.

Memory management failure prevention

As a preventive measure of memory leak, I am a garbage collectorBoehm GCIt is possible to automatically open the area using.If you decide not to free by handwriting, but leave it to automatic GCMemory leak and dangling reference can be almost eliminated.If you are troubled with C, C ++ pointer handling,It is better to consider adoption after considering the trade-off with GC execution efficiency.GCC source includes Boehm GC as standard (used for GNU / Ada),It is good to install GC after C and C ++.

Ultimately, as the language used in the project,Use a language with automatic GC and array subscript range verification.It is possible to create a library to prevent buffer overflow like DJB.However, in general, it is suspicious whether such selfishness can be passed …

Should I use SWAP if memory is enough?

With free you can also observe the amount of swap usage.After starting for a while, it may seem like it’s sticking out to swap.Is this a lack of existing memory?

Some people are concerned about the fact that they are exposed to swap, but they misinterpret the point.I wrote in the previous section that I used memory free space for I / O buffer and cache,Furthermore, the portion with less activity is removed from the memory, pushed out to swap, and diverted to a buffer or cache.In other words, usually using swap meansThe parts of programs and libraries that are rarely used in practice,It shows how the kernel successfully detects and removes it from memory.This alone can not be said to be a shortage of memory.The fact that swap is issued by the free command is just fine.

Note: What is swap?

Even if the memory on the PC is not enough,You can save part of the memory to the disk and continue the calculation.In this way, the mechanism that makes the memory capacity appear larger than it actually is called virtual memory.Therefore, the area for saving memory to disk is called swap.On Linux, swap can also be taken as disk space,You can also reserve (swap file) in the usual file format.

Example with no problems with memory capacity

Memory is over 700M (702888),The surplus is diverted to I / O cache.I use 50M (48628) of swap, but there is no problem at all.

The state of a PC that is regularly used for emacs, TeX, Mozilla, etc.

  1. This machine continues to calculate mathematical programs for more than one month, and the US term of the CPU is constantly high.CPU is “hot”.At the same time, I’m also doing detailed work, so I can see some changes in the procs r (wait for run) section.

  2. As there are not many inputs and outputs, free also shows a margin of 60M (63952).

  3. Even if the CPU id is 0 and procs has no b, w term, swap si, so term has no activity,It indicates that the memory and disk do not have a problem but use the CPU as much as possible.

In this example, if you dare augment it, it will be a CPU.

Well, how do we know the real memory shortage?

Because it is a SAWP to avoid that the program does not run due to lack of memory originally,In such cases, it is better to increase SWAP.Or. Add more memory if funds allow.

So the problem is the difference between the operating speed of the memory and the disk,Compensating for the lack of memory with SWAP slows down the operation.Here, it is necessary to consider the tradeoff between memory cost and execution speed.Let’s look at this point below.

What is bothersome with a lack of memory is that it has the following effects.

  1. There is not enough memory and the program does not work (if so, do you know right away)?

  2. Although it works for a while, its efficiency drops due to intense paging (intensifying swap activity).

Even if paging is “heavy” occasionally and execution efficiency drops, it is better than memory can not run it.It is not necessary to hostile the SWAP operation in such an emergency.If there is a regular swap activity, the problem is whether the operation is slowed by the input / output for that purpose.Make sure that some loss in efficiency is acceptable.

If memory is running short, whether to increase memory,The trade-off between speed and cost required must be taken into account.And, in the first place, is the lack of memory happening,You have to decide whether to solve it by increasing memory.

The memory waste of the process can be checked with free, top, ps, etc. as described above.The remaining memory and the size of the swap are also helpful as the overall memory activity.(Ie in the sense of the potential danger that the above (1), (2) will occur),The highlight in terms of execution efficiency is the amount of disk I / O activity.

swap works by examining the terms si and so with vmstat 30

See disk I / O activity rather than SWAP size.

There is no problem that so temporarily appears when you start running a program.Is it a swap for that?Also, even if it is stationary, if only a small amount of si, so is produced,There is no problem as it is merely active.(Up to this point, the behavior that normally occurs even when memory is available.)

It is also the case that si is stationary,It means that swap is working well (although there is little room for memory),Not very problematic (depends on the speed required …)

An example of a memory shortage

Pay attention to si and so.
       The term swpd is used for the swap area, but the adverse effect on the operating efficiency is easier to determine with si and so.

             

  1. The b section of procs is not working because it is waiting for input / output etc.
                means.
                
                There is no problem that this section appears temporarily.
                If it goes out regularly, check it further.
             

  2.             View the activity of the swap area with si, so.
                
                This may affect the operating efficiency / speed.
                
                In the second half of this example, memory shortage occurs and swap activity is high.
                Since so and si are rising in pairs, you can see that you are struggling to use memory.
                

                   Temporary, no problem.
                   Also, don’t worry if you have the necessary skills.
                   If this is not the case, and service delay is an issue, further inspection is required.
             

  3. View the swap space usage with swpd.
                If a memory shortage occurs in a situation where this overflows, the program will not operate.
                There is no need to think seriously if it does not overflow.
             

  4. Consider bi / bi trends of disk I / O.
                
                This may affect the operating efficiency / speed.
                
                In the first half of this example, the disk input load is high.
                
                You can see the activity in the second half, but it also includes the activity of swap.
             

  5. Network impact.
                
                I do not understand well with vmstat.
                
                This can affect operating efficiency / speed, especially when using NFS.
                Investigate the behavior of NFS and NIS.
             

  6. The large number of CPU ids (idle) means that the CPU speed is not a problem in this situation.
                
                There is also a possibility that the CPU can not be used effectively because of waiting for I / O.
          

      

How much memory and SWAP space should be allocated

      Then, how much real memory and SWAP area should be secured?
      
      Consider the following factors.
      

             

  1.             Active memory when used                The amount of memory required is
                   The reaction rate required by the machine,
                   It should be decided in consideration of SWAP’s (disk I / O) activity trade-off.
                

                   Judging from the activities of si and so of vmstat.
             

  2.             [Parts to be exhaled because activity is dull] <[SWAP]             

                   The purpose is to make the operation light by increasing the free memory.
                   Secure SWAP in the sense of kicking out inactive code.
                

                   If it is operated for a long time, the dull part will be spouted to SWAP.
                   It would be good to consider two to three times more of this exhaled amount.
             

  3.             [Limit of required memory (including safety factor)] <[real memory] + [SWAP]             

                   We evaluate SWAP from the viewpoint of the escape destination when the memory requirement of the running process increases.
                   In this sense, priority is given to keeping the process running anyway.
                

                   Classically, it is recommended to secure SWAP as much as main memory.
                   This is based on the memory required for operation.
                   It can be considered to mean taking into account the double safety factor.
                   As SWAP is actively used, it is permitted to some extent even if the operation is delayed.
                

                   There is also the option of securing a large amount of real memory, including the safety factor.
          


Leave a Reply