Splunk and the ELK Stack use two different approaches to solve the same problem. People typically choose one or the other based on how their organizations are structured and how much time they want to devote to log analytics. Splunk takes a pile of data and allows people to search through the information to extract what they need. ELK requires more work and planning in the beginning, but the value extraction is easier at the end.
Most, if not all, systems and devices in today’s IT environments generate extensive log files that record the minutiae of day-to-day operations: what resources were accessed and by who, activities performed, errors/exceptions encountered by the host, and more. As you can imagine, the volume of log files in any given organization’s infrastructure can quickly become unwieldy. Log management and analysis solutions enable organizations to glean collective, actionable intelligence from this sea of data.
Splunk
Known as the “Google for logfiles,” Splunk is also marketed as a Security Information and Event Management (SIEM) solution, on top of being a log management and analysis platform. SIEM is essentially log management as applied to security: by unifying logfile data gathered from a myriad of systems and devices across an IT environment, operators and infosec professionals can perform higher-order security analyses and assessments regarding the collective state of their systems from a single interface. An abundance of SIEM products exist on the market, but Splunk reigns supreme in this category due to its aforementioned Google-esque search capabilities. The platform uses a proprietary search language called Search Processing Language (SPL) for traversing and executing contextual queries large data sets.
The three key components of Splunk are its forwarder, which pushes data to remote indexers; indexer, which has roles for storing and indexing data and responding to search requests; and search head, which is the front end of the web interface where these three components can be combined or distributed over servers. Splunk also supports the integration of its functionalities in applications via SDKs. Common use cases include operational monitoring, security and user behavior analytics. Splunk is a paid service wherein billing is generated by indexing volume.
ELK/Elastic Stack
The ELK Stack is a set of three open-source products—Elasticsearch, Logstash and Kibana—all developed and maintained by Elastic. Elasticsearch is NoSQL database that uses the Lucene search engine. Logstash is a data processing and transportation pipeline used to populate Elasticsearch with the data (though also it supports other destinations including Graphite, Kafka, Nagios and RabbitMQ). Kibana is a dashboard that works on top of Elasticsearch and facilitates data analysis using visualizations and dashboards.
In addition to the ELK/Elastic stack, each of these technologies is available as a discreet offering from Elastic.
Capability Set
Splunk and ELK/Elastic Stack are powerful, comprehensive log management and analysis platforms that excel in fulfilling the requirements the most demanding enterprise use cases. Both are highly customizable and offers a range of features you’d expect from a competent solution in this category: advanced reporting, robust search capabilities, alerting/notifications, data visualizations, and more.
Loading Data
Shipping data to Splunk is fairly easy. After installation, the forwarders come pre-configured for a wide selection of data sources such as files and directories, network events, windows sources and application logs, and they are used to import data into Splunk.
In the ELK Stack, Logstash is used to ship data from the source to the destination. However, Logstash needs to be configured so that each field is identified before the data is shipped to Elasticsearch. This kind of configuration can be tricky for those who do not work with scripting languages (such as Bash, Python or Ruby), but there is good support online that can be found quite easily.
Visualizations
The Splunk web UI includes flexible controls that allow you to edit and add new components to your dashboard. Management and user controls can be configured differently for multiple users, with each having a customized dashboard. Splunk also supports visualizations on mobile devices with application and visualization components that are easy to customize using XML.
Kibana is the visualization tool in the ELK Stack, and like Splunk, the platform supports the creation of visualizations such as line charts, area arts and tables and the presentation of them in a dashboard. The search filter is always shown above the different views: If a query is used, it is automatically applied on elements of the dashboard. Splunk also has a similar option, but it involves configuration in XML. Still, Kibana does not support user management, but hosted ELK solutions provide it out of the box.
User Management
The ELK Stack provides role-based security as a separate paid tool. Splunk and managed-ELK services offer user management out of the box with user auditing included.
Ease of Use
Both solutions are relatively easy to deploy and use, especially considering each respective platform’s breadth of features and capabilities. That said, Splunk’s dashboards offer more accessible features and its configuration options are a bit more refined and intuitive than ELK/Elastic Stack’s. Additionally, ELK’s user management features are more challenging to use than Splunk’s. On the other hand, AWS offers Elasticsearch as a service that removes much of the difficulty in deploying and managing it.
Community Support
Both Splunk and ELK Stack have large communities of users and supporters. ELK also has its own clear and extensive documentation for each separate tool, making it easy to get started. In addition, Elastic itself offers educational sessions worldwide.
In addition to having good documentation and a forum, Splunk, too, has customer and support platforms that offer various professional services. Splunk’s education program and instructors are available virtually or on site.
Release Rate
Both solutions have seen regular releases over the years: Splunk’s enterprise offering is currently at version 6.5, while ELK/Elastic Stack releases—as a composite platform—are stratified per component. Currently, Elastic Stack (as well as its core components: Kibana, Elasticsearch, Beats, and Logstash) is at version 5.0. Full release histories for Elastic and Splunk are available on the vendors’ websites.
Pricing and Support
Splunk is a proprietary enterprise offering with a high end price tag while ELK/Elastic Stack is a free, open source platform. Despite this, ELK/Elastic Stack’s cost total cost of ownership can be quite substantial as well for expansive infrastructures: hardware costs, price of storage, and professional services can quickly add up (though the aforementioned AWS service can simplify that if cloud-hosting is a viable option). Both Splunk and ELK/Elastic Stack now offer cloud-based, hosted versions for more price-conscious organizations. In terms of support, both ELK/Elastic Stack and Splunk’s support offerings are exceptional.
API and Extensibility
Splunk offers a well-documented RESTful API with over 200 endpoints for accessing every feature in the product as well as SDKs for popular languages. ELK/Elastic Stack’s Elasticsearch was designed from the ground-up as a distributed search and analytics engine using standard RESTful APIs and JSON. It also offers pre-built clients for building custom apps in languages such as Java, Python, .NET, and more.
Vendor Lock-In
Splunk’s high price tag comes with the benefit of offering an overall, well-rounded product. Users might be locked into a vendor, but that one vendor is all that is needed to do nearly anything. The open-source ELK Stack is seemingly free, but it does not provide many functionalities such as alerting out of the box—and it costs money to develop and maintain them.
Companies that Use It
Splunk boasts over 12,000 customers and 80 of the Fortune 100 under its belt: Adobe, BlackRock, Coca-Cola, ING, Tesco, AAA, Staples, among others. Elastic’s customer list is equally impressive, consisting of Ebay, Verizon, Netflix, Cisco, Salesforce, FICO, Facebook Thomson Reuters, to name a few.
Learning Curve
ELK/Elastic Search’s learning curve is surprisingly flat for what it does; Splunk has a moderate learning curve, especially when it comes to building expertise for carrying out more specialized analyses.
At Last!!!
According to Google Trends, the ELK Stack has now overtaken Splunk in terms of the proportion of Google searches. But ELK’s traction does not stop there. As mentioned earlier, Splunk self-reports 12,000 total users. Elasticsearch is reportedly downloaded 500,000 times every single month. In IT departments, then, it is far more likely to meet people who are familiar with ELK than with Splunk, meaning that the adoption rate of the ELK Stack could “snowball” and increase even more in the future whenever ELK users join new companies or teams. People tend to use whatever software they already know or is already being used.
It’s clear that many functionalities are being added to the open-source ELK Stack. This, in turn, is shrinking the gap between it and Splunk. hose features currently found only in Splunk are likely to be added to ELK at some point in time.
