Edit

Architecture

Haystack Components Architecture Diagram

The Haystack architecture is designed on the distributed design principles of building decentralized and decoupled systems. To enable that, we have used Apache Kafka as the nervous system that helps us achieve the following architectural principles in the Haystack server:

• Haystack is Componentized: Haystack includes all of the necessary subsystems to make the system ready to use. But we have also ensured that the overall system is designed in such a way that you can replace any given subsystem to better meet your own needs.
• Haystack is Highly Resilient: There is no single point of failure.
• Haystack is Highly Scalable: We have completely decentralized our system which helps us to scale every component individually.

We provide client components for the client applications or microservices that send trace data to be recorded by the haystack-agent subsystem:

• Clients

We provide the following six subsystems within the server:

• Traces
• Trends
• Collectors
• Pipes
• Dependencies
• Anomaly Detection

Trace data is stored using different services, including:

• Amazon Kinesis Data Firehose to external durable storage.
• Grafana MetricTank for time series metrics.
• Apache Cassandra for raw traces and for trends data.
• AWS ElasticSearch is used as a metadata indexer.

And we provide User Interface components for viewing and analyzing trace data:

• Traces View
• Trends View
• Alerts View
• Service Graph View
• Service Insights View