What is Log Analytics?

Since we started creating computer-generated records, we’ve been trying to analyze those records in bulk. The records are emitted by devices, applications, networks (and more), and are then time-sequenced into logs. Many times these logs are not fully documented or consistently formed across applications or devices, which further builds the case for log analytics.

Log analytics can answer valuable operational analytics questions, like the list below. Imagine if you had real-time answers to these questions:

• Is my infrastructure working?
• What is the latency and error rate?
• What caused my application issue?
• Is there any suspicious authentication activity?
• What data was accessed by this IP address?
• Are there instances of fraud?
• What content/products are my users interested in?
• Which features are used most or least?
• What user are most active and why?

For a more in-depth look at the benefits of log analytics, check out this paper from Omdia.

Primary users for log analytics are DevOps engineers, site reliability staff and enterprise architects.

Your log analytics strategy needs to address data ingest, transformation and enrichment, indexing and sharding strategy, infrastructure planning, and finally data lifecycle and archiving of data. Here are the general steps to take: 

1. First, you need to identify data motions or data ingest. Essentially, you have to figure out an ingest pathway.

2. Second, you need to set up data transformation of log lines or strings. Log analytics many times work with JSON-- something has to transform the data appropriately and enrich if needed.

3. Third, you have to figure out an indexing and shards strategy. Proper creation of indexes is essential.

4. Fourth, you also have to do some infrastructure planning to figure out the instance type and how many of them you need.

5. Finally, to control log size and cost, you need a holistic strategy for data life cycle and archiving.

Deploy and manage with ease

Amazon OpenSearch Service makes it simple to set up and deploy your cluster, while removing the complexity associated with management tasks, such as hardware provisioning, software installing and patching, failure recovery, backups, and monitoring, allowing you to reduce operational overhead and focus on core business requirements.

Get scale and reliability

With up to 3 PB of instance storage in a single domain, Amazon OpenSearch Service lets you easily add or remove instances without any downtime. The service also offers built-in encryption at-rest and in-transit, user authentication, and VPC support, allowing you to keep your data secure.

Integrate easily with other AWS services

Amazon OpenSearch Service offers built-in integrations with other AWS services such as Kinesis Data Firehose, Managed Streaming for Kafka, IoT, CloudWatch Logs, KMS, Cognito, and IAM, so you can securely ingest, analyze, and visualize data from all sources.

Lower your costs

With Amazon OpenSearch Service, you pay only for what you use. There is no upfront fee or usage requirement. With 24x7 monitoring and AWS support, you don’t need a team of Elasticsearch experts to scale, secure, and monitor your infrastructure, resulting in lower total cost of operations.

Real-time application and infrastructure monitoring

Capture and centralize all logs and metrics from your applications and IT silos to get deep visibility into your application and infrastructure stack and ensure uptime. Amazon OpenSearch Service indexes the data, makes it available for analysis in real time, and allows you to visualize the performance metrics in real time using Kibana dashboards.

Expedia Group, one of the world’s leading travel companies, uses Amazon OpenSearch Service for application monitoring. Amazon OpenSearch Service enables Expedia to monitor large volumes of Docker logs cost-effectively, identify and troubleshoot issues in real-time, scale easily to accommodate additional log sources, and offload their operational overhead.  Learn more »

Root-cause analysis

Quickly identify the issues across your environment (servers, code) to reduce the mean-time-to-identification (MTTI) and mean-time-to-resolution (MTTR). With built-in Kibana, Amazon OpenSearch Service lets you search through millions of events and correlate across your applications and infrastructure to quickly diagnose the root-cause of the problem, improving uptime.

Autodesk, a leading provider of 3D design and engineering software, uses AWS services including Amazon OpenSearch Service, Amazon Kinesis Data Firehose, and Amazon Kinesis Data Analytics to build a cost-effective unified logging solution to find and fix application issues faster and improve customer experience.  Learn more »

Clickstream analytics

Get a real-time view of the performance of your web content and user interaction with your applications and websites including user behavior, amount of time spent, popular content, and more. Using Amazon OpenSearch Service and Amazon Kinesis Data Firehose or Amazon Managed Streaming for Kafka, you can aggregate and analyze your clickstream logs effortlessly to gain a deeper understanding of your customers.

Hearst Corporation, a large media company, built a clickstream analytics platform using Amazon OpenSearch Service, Amazon Kinesis Streams, and Amazon Kinesis Firehose to transmit and process 30 terabytes of data per day. With this platform, Hearst is able to make the entire data stream—from website clicks to aggregated data—available to editors in minutes..  Learn more »

Security intelligence and event management (SIEM)

Centralize and analyze events generated across your entire environment including applications, networks, and operating systems to identify any malicious or suspect activity in your network. Amazon OpenSearch Service allows you to index the data as soon as it is ingested, allowing you to analyze data from multiple sources instantly and find and prevent threats faster.