There are several open-source log management systems like Graylog that aim to make it easy to collect, organize, and act on your log data. The product features multiple collectors and input plugins, Elasticsearch-based storage and search of logs, powerful dashboards, and event alerts via email or PagerDuty.
In this article, we’ll take a look at 10 of the best features of log management services to see how they can be applied to specific use cases. Let’s get started!
1. Track down errors in production
It’s one thing to be notified that something went wrong. It’s another thing entirely to be able to quickly pinpoint where, when, and how it happened and rectify it—preventing similar errors in production going forward. This is a major reason why log management tools like Graylog are so useful: they collect logs from all of your different systems in one place and make them accessible for analysis.
2. Search by request, IP address, etc.
Perhaps one of the greatest features is the ability to search log messages by a variety of criteria. This means you can search for specific words or phrases, and only get backlog messages that mention them.
Perhaps you’re looking for all log messages containing particular terms (for example, user and authentication), or perhaps you want to know when users on a certain IP address accessed your system (for example, 192.168.1.254). Whatever your reasons may be, searching in Graylog allows you to find it easily and quickly! But there’s more! Read on for some lesser-known ways to search.
3. Get insights into your website visitors
The built-in search and reporting functionality makes it easy to get valuable insights into your website visitors. Who are they? Where do they come from? What did they do on your site? Which pages, files, or post types are most popular with them?
All of these questions can be answered with a single click—and since everything is available in real-time, you can watch trends unfold before your eyes. Automated alerts allow you to catch issues early and prevent problems before they become major issues. Read on to find out how you can get started with these features right away!
4. Integrate with other systems
If you’re monitoring a lot of servers, log management can be used to provide a centralized place to collect and analyze data. If your organization uses many tools that produce logs, then you can integrate them as well.
For example, if you use Amazon Web Services (AWS), then you can use its agent to send logs to Graylog so they don’t get lost in AWS CloudWatch. This allows you to gain deeper insights into things like server utilization and application performance when viewed alongside metrics from across your entire cloud environment.
5. Deploy it easily with Docker
Docker containers are a simple and flexible way to deploy applications. And that’s why we chose Docker to power Logentries on-premises log management solution for Linux. Deploying Logentries with Docker is as easy as spinning up your first container!
Check out a quick start guide for details on how to get up and running with Logentries using Docker. In no time at all, you can be viewing and analyzing your logs through a Web UI or customizing rules to watch them in real-time in an ELK stack powered by Logentries—and it’s free!
6. Receive alerts
Log management allows you to configure alerting rules based on log messages. Using these, you can be notified via email, pagerduty or hipchat whenever certain events occur in your application’s logs. This can be used as a substitute for or in addition to existing monitoring systems such as nagios or zabbix.
For example: if you are using graylog with elasticsearch, you could create an alert rule that alerts you when there are more than 10k documents waiting for indexing by elasticsearch after 15 minutes of inactivity. This will not only notify you immediately if your system is failing to process data but also gives information on what kind of processing problem it is having.
7. Analyze data in real time with Kibana
Log management is all about analyzing data as it’s collected in real time. Kibana, a visualization tool that comes with a tool setup, allows you to manipulate and explore data from your logs using dashboards and graphs.
For example, you can see how many connections refused errors your server receives over a given period of time or create a graph of latency measurements for an important API endpoint. The possibilities are endless—and only limited by your imagination!
8. Scale to 100,000 events per second
For organizations with a high volume of log data, it’s critical to find a solution that can scale. Many tools are built to grow along with your company that are confident that their service can handle any amount of logs you can throw at us.
If your organization has 100,000+ events per second in production, don’t worry! Check out their special case study on how one customer managed their growth. Find out more about their high availability architecture.
9. Use Elasticsearch for search indexing (instead of Lucene )
Elasticsearch and Lucene are both full-text search libraries. That said, they have many differences and offer different functionality. Elasticsearch is designed to scale horizontally, while Lucene is designed to be fast when querying a single document.
Elasticsearch uses MapReduce for complex searching on distributed systems; Lucene does not use MapReduce. For more details on how these technologies differ, see Choosing an Elasticsearch or Solr Index Type.
10. A truly open source project
There are many open source tools meaning anyone can access and contribute to their code. Unlike many other log management platforms out there, you don’t have to pay a licensing fee or deal with artificial limitations.
It also means that if you want to tweak one of your logs to show things differently, you can do it by modifying just one line of code—no advanced coding skills required. When you know how something works, it’s easier to make changes than when using an opaque black box system. That knowledge allows developers and operators alike to be more nimble and react quickly to new threats.
Summary
As an open-source log management platform, there are many tools known for their flexibility and scalability. Let’s look at 10 things you probably didn’t know about log management tools that are covered in our list of frequently asked questions.
