Monthly Archives: June 2018
How to Check Disk Space on Linux from the Command Line | Linux.com
Quick question: How much space do you have left on your drives? A little or a lot? Follow up question: Do you know how to find out? If you happen to use a GUI desktop (e.g., GNOME, KDE, Mate, Pantheon, etc.), the task is probably pretty simple. But what if you’re looking at a headless server, with no GUI? Do you need to install tools for the task? The answer is a resounding no. All the necessary bits are already in place to help you find out exactly how much space remains on your drives. In fact, you have two very easy-to-use options at the ready.
In this article, I’ll demonstrate these tools. I’ll be using Elementary OS, which also includes a GUI option, but we’re going to limit ourselves to the command line. The good news is these command-line tools are readily available for every Linux distribution. On my testing system, there are a number of attached drives (both internal and external). The commands used are agnostic to where a drive is plugged in; they only care that the drive is mounted and visible to the operating system.
With that said, let’s take a look at the tools.
df
The df command is the tool I first used to discover drive space on Linux, way back in the 1990s. It’s very simple in both usage and reporting. To this day, df is my go-to command for this task. This command has a few switches but, for basic reporting, you really only need one. That command is df -H. The -H switch is for human-readable format. The output of df -H will report how much space is used, available, percentage used, and the mount point of every disk attached to your system (Figure 1).
What if your list of drives is exceedingly long and you just want to view the space used on a single drive? With df, that is possible. Let’s take a look at how much space has been used up on our primary drive, located at /dev/sda1. To do that, issue the command:
df -H /dev/sda1
The output will be limited to that one drive (Figure 2).
You can also limit the reported fields shown in the df output. Available fields are:
-
source — the file system source
-
size — total number of blocks
-
used — spaced used on a drive
-
avail — space available on a drive
-
pcent — percent of used space, divided by total size
-
target — mount point of a drive
Let’s display the output of all our drives, showing only the size, used, and avail (or availability) fields. The command for this would be:
df -H --output=size,used,avail
The output of this command is quite easy to read (Figure 3).
The only caveat here is that we don’t know the source of the output, so we’d want to include source like so:
df -H --output=source,size,used,avail
Now the output makes more sense (Figure 4).
du
Our next command is du. As you might expect, that stands for disk usage. The du command is quite different to the df command, in that it reports on directories and not drives. Because of this, you’ll want to know the names of directories to be checked. Let’s say I have a directory containing virtual machine files on my machine. That directory is /media/jack/HALEY/VIRTUALBOX. If I want to find out how much space is used by that particular directory, I’d issue the command:
du -h /media/jack/HALEY/VIRTUALBOX
The output of the above command will display the size of every file in the directory (Figure 5).
So far, this command isn’t all that helpful. What if we want to know the total usage of a particular directory? Fortunately, du can handle that task. On the same directory, the command would be:
du -sh /media/jack/HALEY/VIRTUALBOX/
Now we know how much total space the files are using up in that directory (Figure 6).
You can also use this command to see how much space is being used on all child directories of a parent, like so:
du -h /media/jack/HALEY
The output of this command (Figure 7) is a good way to find out what subdirectories are hogging up space on a drive.
The du command is also a great tool to use in order to see a list of directories that are using the most disk space on your system. The way to do this is by piping the output of du to two other commands: sort and head. The command to find out the top 10 directories eating space on a drive would look something like this:
du -a /media/jack | sort -n -r | head -n 10
The output would list out those directories, from largest to least offender (Figure 8).
Not as hard as you thought
Finding out how much space is being used on your Linux-attached drives is quite simple. As long as your drives are mounted to the Linux system, both df and du will do an outstanding job of reporting the necessary information. With df you can quickly see an overview of how much space is used on a disk and with du you can discover how much space is being used by specific directories. These two tools in combination should be considered must-know for every Linux administrator.
And, in case you missed it, I recently showed how to determine your memory usage on Linux. Together, these tips will go a long way toward helping you successfully manage your Linux servers.
Learn more about Linux through the free “Introduction to Linux” course from The Linux Foundation and edX.
5 Commands for Checking Memory Usage in Linux | Linux.com
The Linux operating system includes a plethora of tools, all of which are ready to help you administer your systems. From simple file and directory tools to very complex security commands, there’s not much you can’t do on Linux. And, although regular desktop users may not need to become familiar with these tools at the command line, they’re mandatory for Linux admins. Why? First, you will have to work with a GUI-less Linux server at some point. Second, command-line tools often offer far more power and flexibility than their GUI alternative.
Determining memory usage is a skill you might need should a particular app go rogue and commandeer system memory. When that happens, it’s handy to know you have a variety of tools available to help you troubleshoot. Or, maybe you need to gather information about a Linux swap partition or detailed information about your installed RAM? There are commands for that as well. Let’s dig into the various Linux command-line tools to help you check into system memory usage. These tools aren’t terribly hard to use, and in this article, I’ll show you five different ways to approach the problem.
I’ll be demonstrating on the Ubuntu Server 18.04 platform. You should, however, find all of these commands available on your distribution of choice. Even better, you shouldn’t need to install a single thing (as most of these tools are included).
With that said, let’s get to work.
top
I want to start out with the most obvious tool. The top command provides a dynamic, real-time view of a running system. Included in that system summary is the ability to check memory usage on a per-process basis. That’s very important, as you could easily have multiple iterations of the same command consuming different amounts of memory. Although you won’t find this on a headless server, say you’ve opened Chrome and noticed your system slowing down. Issue the top command to see that Chrome has numerous processes running (one per tab – Figure 1).
Chrome isn’t the only app to show multiple processes. You see the Firefox entry in Figure 1? That’s the primary process for Firefox, whereas the Web Content processes are the open tabs. At the top of the output, you’ll see the system statistics. On my machine (a System76 Leopard Extreme), I have a total of 16GB of RAM available, of which just over 10GB is in use. You can then comb through the list and see what percentage of memory each process is using.
One of the things top is very good for is discovering Process ID (PID) numbers of services that might have gotten out of hand. With those PIDs, you can then set about to troubleshoot (or kill) the offending tasks.
If you want to make top a bit more memory-friendly, issue the command top -o %MEM, which will cause top to sort all processes by memory used (Figure 2).
The top command also gives you a real-time update on how much of your swap space is being used.
free
Sometimes, however, top can be a bit much for your needs. You may only need to see the amount of free and used memory on your system. For that, there is the free command. The free command displays:
-
Total amount of free and used physical memory
-
Total amount of swap memory in the system
-
Buffers and caches used by the kernel
From your terminal window, issue the command free. The output of this command is not in real time. Instead, what you’ll get is an instant snapshot of the free and used memory in that moment (Figure 3).
You can, of course, make free a bit more user-friendly by adding the -m option, like so: free -m. This will report the memory usage in MB (Figure 4).
Of course, if your system is even remotely modern, you’ll want to use the -g option (gigabytes), as in free -g.
If you need memory totals, you can add the t option like so: free -mt. This will simply total the amount of memory in columns (Figure 5).
vmstat
Another very handy tool to have at your disposal is vmstat. This particular command is a one-trick pony that reports virtual memory statistics. The vmstat command will report stats on:
-
Processes
-
Memory
-
Paging
-
Block IO
-
Traps
-
Disks
-
CPU
The best way to issue vmstat is by using the -s switch, like vmstat -s. This will report your stats in a single column (which is so much easier to read than the default report). The vmstat command will give you more information than you need (Figure 6), but more is always better (in such cases).
dmidecode
What if you want to find out detailed information about your installed system RAM? For that, you could use the dmidecode command. This particular tool is the DMI table decoder, which dumps a system’s DMI table contents into a human-readable format. If you’re unsure as to what the DMI table is, it’s a means to describe what a system is made of (as well as possible evolutions for a system).
To run the dmidecode command, you do need sudo privileges. So issue the command sudo dmidecode -t 17. The output of the command (Figure 7) can be lengthy, as it displays information for all memory-type devices. So if you don’t have the ability to scroll, you might want to send the output of that command to a file, like so: sudo dmidecode –t 17 > dmi_infoI, or pipe it to the less command, as in sudo dmidecode | less.
/proc/meminfo
You might be asking yourself, “Where do these commands get this information from?”. In some cases, they get it from the /proc/meminfo file. Guess what? You can read that file directly with the command less /proc/meminfo. By using the less command, you can scroll up and down through that lengthy output to find exactly what you need (Figure 8).
One thing you should know about /proc/meminfo: This is not a real file. Instead /pro/meminfo is a virtual file that contains real-time, dynamic information about the system. In particular, you’ll want to check the values for:
-
MemTotal
-
MemFree
-
MemAvailable
-
Buffers
-
Cached
-
SwapCached
-
SwapTotal
-
SwapFree
If you want to get fancy with /proc/meminfo you can use it in conjunction with the egrep command like so: egrep –color ‘Mem|Cache|Swap’ /proc/meminfo. This will produce an easy to read listing of all entries that contain Mem, Cache, and Swap … with a splash of color (Figure 9).
Keep learning
One of the first things you should do is read the manual pages for each of these commands (so man top, man free, man vmstat, man dmidecode). Starting with the man pages for commands is always a great way to learn so much more about how a tool works on Linux.
Learn more about Linux through the free “Introduction to Linux” course from The Linux Foundation and edX.
KDE Plasma 5.13 Is Here » Linux Magazine
The KDE Project has announced the release of Plasma 5.13, the latest version of its desktop environment. KDE is known for its modular design and under-the-hood customization. However, at times these benefits come at the cost of resource efficiency. But as KDE is targeting mobile devices, this release takes advantage of that work and has been optimized to run smoothly on under-powered ARM laptops, high-end gaming PCs, and everything in between. Resource efficiency also means that on powerful machines, more resources will be free for applications instead of being consumed by the desktop itself.
Web browsers are the gateway to the Internet; Plasma 5.13 comes with browser integration that allows users to monitor and control supported browsers, including Chrome/Chromium and Firefox, from the desktop widget. Users will be able to play and pause media playing in web browsers, offering users better control over not only their own entertainment, but also to control annoying autoplaying videos embedded on websites.
The community has also improved the KDE Connect experience; users can now send links directly to phone using KDE Connect. The Media Control Widget has been redesigned with added support for the MPRIS specification, which means media players can now be controlled from the media controls in the desktop tray or from a phone using KDE Connect.
On the security side, Vaults, Plasma’s storage encryption utility, includes a new CryFS backend,
better error reporting, a more polished interface, and the ability to remotely open and close vaults via KDE Connect.
KDE already had good multi-monitor support, where you could even choose a customized layout for each monitor. The 5.13 release makes it easier to connect external monitors. When a new external monitor is connected, a dialog pops up offering the option to control the position of the additional monitor in correlation to the primary monitor.
The desktop has also received some visual upgrades, from the login screen to icons. Plasma 5.13 will appear in different distributions depending on their own release cycle, but users can test the latest release with KDE’s own distribution called “neon”. openSUSE Tumbleweed and Arch Linux will be among the first to offer this release.
Source:
GitLabs Drops Pricing After Microsoft, GitHub A… » Linux Magazine
As the news broke that Microsoft was acquiring GitHub, panicked users started to move their accounts to GitLabs, a fully open source implementation of Linus Torvalds Git.
While many leading figures of the open source world argues that GitHub is actually now in a more accountable and reliable position compared to earlier, because Microsoft will be treading carefully so as to not stain the positive image the company has been building with the open source community.
However, that didn’t stop users from move away from GitHub. Sensing an opportunity, GitLabs dropped pricing for its self-hosted GitLab Ultimate plan and its hosted Gold plan; both plans are now available for free to open source projects and educational institutions.
In an interview to Frederic Lardinois of TechCrunch, GitLab CEO Sid Sijbrandij said, “Most education and open source projects don’t have access to enhanced security or performance management tools for their software projects. At GitLab, we are happy to have achieved a level of success that allows us to extend the full set of features to these important communities by offering GitLab Ultimate & GitLab Gold plans for free.”
A caveat, these prices have been dropped, but these users won’t get any commercial support form GitLabs that paying users get.
How Erasure Coding is Evolving
Data resiliency is at a crossroads. Traditional SAN storage solutions than run on Redundant Array of Independent Disks (RAID) are creaking under the strain of new data demands. While striping, mirroring, and parity in RAID implementations provide various degrees of protection, the cost of resiliency, recovery timings, and RAID’s recovery process vulnerability issues are all paving the way for alternatives.
One option is erasure coding , which is distinctly different than other hardware-based systems. EC is an algorithm-based implementation that’s is not tied down to any specific hardware. It breaks the data into fragments, augments and encodes them with redundant pieces of information, and then distributes encoded fragments across disks, storage nodes, or locations. With erasure coding, data which becomes unreadable on a node can still be reconstructed using information about the data stored elsewhere.
Unlike RAID, the EC does not require a specialized hardware controller and provides better resiliency. More importantly, it provides protection during the recovery processes. Depending on the degree of resiliency, complete recovery is even possible when only half of the data elements are available — that’s a major advantage over RAID. Compared with mirroring, EC also consumes less storage. The down side, however, is that EC is CPU-intensive and can cause latency issues.
Storage efficiency vs. fault tolerance
Erasure coding is most often rendered using Reed-Solomon (RS) codes. For those familiar with RS codes, two performance metrics matter: storage efficiency and fault tolerance. EC involves a trade-off between the two. Storage efficiency is an indicator of additional storage required to assure resiliency, whereas fault tolerance is an indicator of the possibility of recovery in the event of element failures.
These metrics are inversely proportional to one another; more fault tolerance reduces the storage efficiency. That is to say, the more distributed, and therefore geographically widespread the data is stored, the more latency occurs as the time required to recall from different locations or systems is greater.
Hyperscale data centers pose fresh challenges for data resiliency in terms of node failures and degraded reads. Modern erasure code algorithms have evolved to include local regeneration codes, codes with availability, codes with sequential recovery, coupled layer MSR codes, selectable recovery codes, and others that are highly customized.
Acceleration and off-loading
Erasure codes are compute intensive and it has become necessary to offload that compute from the main CPU. Research looking into options for optimizing various aspects is well underway in academia and in industry. Innovations in data center hardware are promising too. Whether virtual or bare metal, there is a greater probability of freeing up computation resources here, like GPU and FGPA.
One of the requirements of GPU-based acceleration is parallelization of the EC algorithms. Parallelization is based on the concept of parallel computing, when multiple processes are executed concurrently and the modern resiliency codes have some cases of the vector codes. These vector approaches make it possible to leverage GPU cores and high-speed on core memory (like Texture Memory) to achieve parallelism.
Fabric acceleration is another trend in EC off-loading. Next-generation host channel adapters (HCA) offer calculation engines, making full use of features like RDMA and verbs. Encode and transfer operations are handled in HCA. With RDMA, it proposes more acceleration for storage clusters.
Data resiliency, compression, and deduplication advances are evolving at breakneck speed. It is an exciting time for erasure coding: extreme low latencies of NVMe technologies, tighter integration of storage with application characteristics, and newer virtualization options are opening up a myriad of use cases. As traditional RAID systems reach their data resiliency limits, data center and storage professionals can consider systems based on erasure coding as a strong option to provide resiliency, protect data during recovery, and minimize storage requirements.
Dinesh Kumar Bhaskaran, Director of Technology and Innovation at Aricent, has more than 15 years of experience in embedded and enterprise storage technologies. He also works for the innovation group leading efforts in the field of hyper converged infrastructure. His areas of interest include Erasure Coding, Heterogeneous Computing and Operating Systems.
